SAM-Guided Masked Token Prediction for 3D Scene Understanding
Authors: Zhimin Chen
Liang Yang
Yingwei Li
Longlong Jing
Bing Li
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D self-supervised learning
Multi-modal Representation Learning
Masked autoencoders
Knowledge distillation
Foundation models
Primary area: other

Abstract: Foundation models have significantly enhanced 2D task performance, and recent works like Bridge3D have successfully applied these models to improve 3D scene understanding through knowledge distillation, marking considerable advancements. Nonetheless, challenges such as the misalignment between 2D and 3D representations and the persistent long-tail distribution in 3D datasets still restrict the effectiveness of knowledge distillation from 2D to 3D using foundation models. To tackle these issues, we introduce a novel SAM-guided tokenization method that seamlessly aligns 3D transformer structures with region-level knowledge distillation, replacing the traditional KNN-based tokenization techniques. Additionally, we implement a group-balanced re-weighting strategy to effectively address the long-tail problem in knowledge distillation. Furthermore, inspired by the recent success of masked feature prediction, our framework incorporates a two-stage masked token prediction process in which the student model predicts both the global embeddings and token-wise local embeddings derived from the teacher models trained in the first stage. Our methodology has been validated across multiple datasets, including SUN RGB-D, ScanNet, and S3DIS, for tasks like 3D object detection and semantic segmentation. The results demonstrate significant improvements over current state-of-the-art self-supervised methods, establishing new benchmarks in this field.

SAM-Guided Masked Token Prediction for 3D Scene Understanding
Zhimin Chen' Clemson University
Liang Yang² The City University of New York
Yingwei Li
Johns Hopkins University
Longlong Jing The City University of New York
Bing Li Clemson University
Abstract
Foundation models have significantly enhanced 2D task performance, and recent works like Bridge3D have successfully applied these models to improve 3D scene understanding through knowledge distillation, marking considerable advancements. Nonetheless, challenges such as the misalignment between 2D and 3D represen- tations and the persistent long-tail distribution in 3D datasets still restrict the effectiveness of knowledge distillation from 2D to 3D using foundation models. To tackle these issues, introduce novel SAM-guides tokenization method that seamlessly aligns 3D transformer structures with region-level knowledge distilla- tion, replacing the traditional KNN-based tokenization techniques. Additionally, we implement a group-balanced re-weighting strategy to effectively address the long-tail problem in knowledge distillation. Furthermore, inspired by the recent success of masked feature prediction, our framework incorporates a two-stage masked token prediction process in which the student model predicts both the global embeddings and the token-wise local embeddings derived from the teacher models trained in the first stage. Our methodology has been validated across multiple datasets, including SUN RGB-D, ScanNet, and S3DIS, for tasks like 3D object detection and semantic segmentation. The results demonstrate significant improvements over current offte-of-thhear self-supervised methods, establishing new benchmarks in this field.
Introduction
3D computer vision plays a critical role domains such as autonomous driving and robotics. Despite its importance, this field faces significant challenges in acquiring and annotating 3D data due to the high costs and complex technical requirements involved. These challenges have led to notable scarcity of large-scale annotated datasets. To address these issues, there has been growing shift towards self-supervised learning (SSL) strategies, including contrastive learning and masked autoencoders (MAE), which aim improve the learning efficiency of networks and reduce reliance on labeled data. Recently, the success of 2D foundation models like Contrastive Vision- Vision-Languaee Language Pre-training (CLIP) [41] and Segment Anything (SAM) [29] has led significant progress image understanding. However, large-scale 3D foundation models have not yet been proposed, primarily due to the scarcity of 3D datasets. Therefore, leveraging these powerful 2D foundation models for 3D scene understanding via self-supervised learning remains an open question.
Recent works, such as CLIP2Scene [8], Seal [32], and Bridge3D [10], have made significant progress in enhancing 3D scene understanding through the use of foundation models. CLIP2Scene effectively integrates CLIP with 3D models by implementing pixel-to-point distillation via foundation models.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Patch based 2D tokenization method.
(b) KNN-based 3D tokenization method.
(c) Proposed SAM-guided 3D tok- enization method.
Figure 1: The comparison of tokenization methods. In Section 3.2, we present a detailed compari- son of our proposed tokenization method to the previous KNN-based approach. As shown in the red circle, the KNN-based method may inadvertently group points from different SAM regions into the same tokens, leading potential confusion within the 3D network. In contrast, our method effectively employs SAM masks in tokenization to ensure seamless region-level knowledge distillation, thereby avoiding these issues.
Scal distills the knowledge from 2D foundation models into the 3D network for semantic segmentation. Bridge3D introduces an innovative pre-training strategy for 3D models by utilizing features, semantic masks, and captions derived from foundation models. However, significant challenges still remain in leveraging foundation models for 3D scene understanding. Specifically, while CLIP2Scene employs point-to-text contrastive learning, it does not utilize critical region-specific information, which is essential for dense representation learning. Seal leveraged the 3D U-Net as backbone which struggles with scalability and flexibility, making it less effective for scaling and handling tasks such detection. Bridge3D attempts to address this by using SAM-generated masks to distill vision and language representations point tokens the regional level. However, illustrated in Figure I, both Bridge3I and earlier 3D transformer- based methods employ KNN-based point tokenization strategies that can result in information conflicts during SAM-guided region-level knowledge distillation. This conflict arises when points from different SAM regions are grouped into the same 3D tokens, thereby confusing the 3D network. Furthermore, both CLIP2Scene and Bridge3D do not take into account the inherently long-tail property of 3D datasets. Giving equal weight to all samples causes the model to be predominantly driven by gradients from a few over-represented samples, leading to poor performance on under-represented samples.
To overcome these challenges, we propose a SAM-guided masked token prediction method that facilitates region-level 2D-to-3D knowledge distillation using foundation models. Unlike traditional 3D transformer methods that rely on KNN for tokenization, our approach employs masks obtained from SAM to tokenize points. This strategy effectively prevents conflicts between different regions and points, ensuring a seamless integration of point tokenization with region-level knowledge distillation. Additionally, SAM masks improve the representation of homogeneous neighboring points by more effectively leveraging boundary regularities compared to KNN-based methods. Furthermore, to address the issue of representation imbalance, we introduce a group- balanced re- re-weighting strategy that adjusts the distillation loss weights between 2D and 3D representations the region level. In the self-supervised phase, where labels are absent, we utilize well-trained 2D foundation models to cluster region-level 2D representations using K-means, assigning pseudo-labels based on their cluster index. During training, we enhance the weights for tail groups while reducing them for head groups. Inspired by the recent success of masked feature prediction [56] in cross-modality learning, we introduce two-stage masked token prediction framework. In the first stage, we perform dense region-level knowledge distillation using the SAM-guided tokenization method to transfer well-learned knowledge from the foundation model to the 3D network. In the second stage, have the student model predict both the instance- global embedding and the token-wing local embeddings obtained from the teacher model in the first stage based visible 3D input patches. This approach ensures that the student model learns well-aligned and contextualized local and global representations, thereby improving its performance downstream tasks.
We validated our methodology across multiple datasets and tasks, including SUN RGB-D [46) and ScanNet [14] for 3D object detection, and S3DIS [4] and ScanNet [14] for 3D semantic segmentation.

---

Our approach outperforms current state-of oaftte-of-hhea-aee art self-supervised learning methods, underlining the effectiveness of our proposed framework.
The key contributions of our work are summarized as follows:
We introduce a novel two-stage SAM-guided masked token prediction framework that leverages foundation models for 3D scene understanding.
We present a group-halanced re-weighting method for long-tail representation distillation and a SAM-guided tokenization method to seamlessly align 2D and 3D region-level features.
Extensive experiments have been conducted to demonstrate the significance of our approach in various 3D downstream tasks.
2 Related Work
3D Self-supervised Pre-training. The field of self-supervised learning for point clouds has witnessed substantial advancements, with researchers exploring a variety of pre-training strate- gies to enhance the transferability and initialization quality of networks for downstream tasks [1, 21, 11, 57, 28, 19]. These strategies range from learning the relative positions of points [43] deploying multiple pretext tasks [22] and employing contrastive learning approaches [17, 27, 42, 49, 2, 28, 20, 18). Innovatively, Info3D [42] applies InfoMax principles and con- trastive learning to 3D shapes, enhancing feature extraction from complex geometries. PointContrast [49] performs point-level contrastive learning across transformed views of a single point cloud, pro- moting robustness spatial alterations. Meanwhile, the work by Zhang [55] contrasts instance-level representations derived from different architectural processes within identical scenarios. Additionally, CrossPoint [2] pioneers modal contrastive framework that establishes 3D-2D correspondences. capitalizing on the complementary attributes of point clouds and images.
Masked Autoencoder To enhance masked image modeling, the Masked Autoencoder [23] (MAE) was initially introduced for 2D images, utilizing asymmetric encoder-decode transformer ar- chitecture [16, 6]. This process starts with an encoder that receives a randomly masked image to extract high-level latent representations. A lightweight decoder then processes these representations, reconstructing the raw RGB pixels of the masked patches. Demonstrating exceptional performance across various downstream tasks, MAE has inspired a range of innovative adaptations. Expanding to 3D data, some adaptation methods [37, 12, 53] have been proposed to apply MAF-style pre- training to 3D point clouds. These methods involve sampling visible point tokens for the encoder and reconstructing masked 3D coordinates with the decoder. However, these 3D MAE applications have predominantly focused on masked point reconstruction. Recent studies [5, 56] have shown that masked feature prediction can be a more effective strategy for representation learning. Building on this insight, our work introduces a two-stage framework specifically designed to enable masked token prediction in 3D scene understanding, aiming to enhance the learning efficiency and applicability of MAE in complex 3D environments. This novel approach promises to push the boundaries of how deep learning models perceive and interpret three-dimensional data.
Multi-modality Learning for 3D Scene Understanding Numerous studies have explored knowl- edge transfer from pre-trained 2D foundation models to 3D representations at the object level [24, 25, 54, 52, 38]. For comprehensive scene understanding. SlidR [44] initially employs the super-pixel technique define regions and subsequently utilizes the InfoNCE loss for region-level contrastive learning hetween point cloud and 2D representations. The CLIP2Scene approach [8] leverages the MaskClip model [59] to generate dense semantic predictions. However, it lacks the capability to produce instance-specifte semantic results, which is crucial for distilling object-level visual representations into 3D models. Bridge3D [10] proposes an innovative pre-training strategy for 3D models using features, semantic masks, and captions derived from foundation models. It integrates region-leve knowledge distillation with a masked autoencoder for 3D scene understanding. achieving -fate-offnn of-te--f--heaar performance. Despite these advancements, the KNN- based tokenization method employed in Bridge3D and other existing 3D transformer-basee methods faces challenges. Specifically, the mismatch between KNN grouping and predefined mask regions can lead to represen- tational conflicts, ultimately degrading performance. To address these limitations, we innovatively propose a SAM-guided tokenization method that scamlessly integrates region-leved level distillation with
3

---

Stage Region-level Dense Distillation
Image
Regional 2D Representation
Regional Distilition
Group Assign
Encoder
Point Clouds
Group Balanced Weight
SAM-guided 3D Tokenization
Regional 3D Representation
3D Encoder
Projector
Stage 2, Masked Token Prediction
Iatance-level
Taken-level
Distilation
Distillation
OOPO
Teacher Encoder
Point Clouds
SAM Tokenization
& Mocking
Encoder
Decoder
Figure 2: Overall framework of the proposed method. Our method introduces two-stage masked token prediction framework for learning from foundation models. In the first stage, we input complete point clouds and leverage SAM masks to guide the point cloud tokenization, thereby seamlessly aligning the 2D and 3D region-level features for dense prediction. A group-balanced weight is applied during distillation to prevent bias towards the head representations. In the second stage, freeze the models trained in the first stage and have the student models predict instance-level features and masked tokens obtained from the teacher models.
plain transformer architecture. This method ensures coherent region-level knowledge transfer and enhances the overall efficacy of the learning process in 3D scene understanding.
3 Methodology
In this section, we outline our methodology, starting with how we utilize foundation model SAM [29] to obtain masks offline. We then describe our approach to tokenize point clouds using SAM instead of KNN to address the misalignment between 2D and 3D representations. Next, we introduce group-balanced re-weighting strategy to address the long-tail representation issue in knowledge distillation. Finally, we detail our two-stage masked token framework, which facilitates the model in learning well-aligned and contextualized representations through process of two-stage masked token prediction.
3.1 Mask Generation
To enable region-level distillation, we utilized the foundation model SAM [29] to generate masks within visual images. SAM- SAM-generated masks provide comprehensive coverage of both the object and its surrounding context. This integration enables us to obtain a cohesive set of segmentation masks O,,..., .O. To establish a precise correspondence between mask-level visual features and point tokens (1: Pi), we align the point cloud tokens with the respective SAM masks, where and p; represent paired image and point features, respectively. This process conducted offline, and the resulting labels are stored locally for easy access during the self-supervised training phase.

---

3.2 SAM-guided Point Tokenization
Recent state-of-the-art method Bridge3D [10] adapts the 3D plain transformer architecture for knowledge distillation from foundation models. Like other 3D transformer-baset methods, Bridge3D directly tokenizes point clouds using farthest point sampling (FPS) and K-nearest neighbors (KNN) algorithms. Specifically. given a point cloud X R& with N points, FPS used to select n centroid points (CT) for forming point patches. Subsequently, KNN determines the k nearest points to each centroid, forming the corresponding token P.
To enable efficient distillation from 2D to 3D using foundation models, Bridge3D proposes a region- level distillation strategy. However, we find that this strategy cannot effectively align 2D and 3D information when using KNN-based point tokenization. As depicted in Figure I, KNN-based point tokenization strategies can group points from different SAM regions into the same 3D tokens. This misalignment causes information conflict, which confuses the 3D network and degrades the distillation performance.
To address this challenge, we introduce a novel SAM-guided point patch generation method tailored for multi- multi-modality region-level knowledge distillation. We start by projecting the 3D point cloud onto corresponding 2D images. Then, we assign points to tokens based on their positions within the SAM-defined regions in the 2D images. Each patch's 's centroid is calculated the average position of all points within that patch. Features are then extracted using PointNet, ensuring that each token' ' representation is both cohesive and regionally consistent. This methodology not only enhances the alignment between 3D points and their corresponding 2D regions but also significantly improves the performance of our knowledge distillation framework by leveraging boundary regularities provided by SAM.
3.3 Dense Feature Distillation
To enhance the distillation of rich representations from 2D to 3D, we build the methodologies proposed in Bridge3D [10]. utilizing region proposals from the SAM vision foundation model. Our method primarily differs from Bridge3D in our approach aligning 2D and 3D representations. Unlike Bridge3D, which employs the traditional KNN method to generate point tokens, often resulting in the aggregation of points from disparate regions as illustrated in Figure 1, our method uses the SAM model to guide point cloud tokenization. This ensures a one-to-one correspondence between point tokens and region-level 2D features, avoiding the representation conflicts that impair 3D understanding in the Bridge3D approach. This targeted tokenization results in a comprehensive set of segmentation masks O I..... ON. each enriched with its corresponding textual narrative, thereby providing deeper contextual dataset.
For detailed feature extraction, define EgD the trainable 3D network and Eyp as the frozen pre-trained 2D encoder. The 3D network processes the point cloud, while the 2D encoder manages the corresponding images. These components generate 3D point token features H R and 2D image pixel features I E R where M represents the number of regions, equivalent to the segmentation masks produced by SAM, and I. denotes the feature dimension; h and height and width of the image features, respectively. We project the point token features to 2D space via a projection layer obtain projected 3D features FAD. We then pool pixel representations within the same SAM-generated region to derive region-level 2D representations F2D E _ Simultaneously,, we establish correspondences between each 3D token and its matching 2D regional representation (H,, M where H, and F. are the paired 3D and 2D regional features, ensuring a direct and meaningful alignment between the modalities. This setup allows for robust region-level feature-dense distillation, effectively training our model to better understand and interpret complex 3D scenes. The distillation process is formulated follows:
the
F2D = (1)) O, iEO;
(1)
FOD = Proj(H)
(2)
5

---

Where H; represents point tokens, Proj is the projection layer. The objective function is as follows:
1 M
Edistill
Σ (F FsD,i)
M
The L is the smooth L loss.
3.4 Group Balanced Re-weighting
Training models on 3D datasets, which are inherently highly imbalanced, presents significant chal- lenges. Directly applying approaches that are not specifically designed for imbalanced datasets often results sub-optimal network performance and thus fails to deliver satisfactory outcomes. To address this issue, recent studies, noted by Cui et al. [13], Yu al. [50], and Alshammari et al. [3], have introduced
loss strategies. These strategies involve recalibrating the weights in the loss function to focus more underrepresented (tail) classes and reduce the emphasis on overrepresented (head) classes. Such adjustments aim to establish a more equitable training environment, enhancing fairness and boosting the robustness of the model.
In our 2D to 3D pre- training tasks, the lack of explicit labels complicates the identification of head and tail representations within the data. To tackle this challenge of imbalance in the representation learning stage, we propose a prototype-Ieve re-weighting method. Leveraging the discriminative features provided by foundation models, we can directly cluster these features and use their cluster indices as pseudo-labels. Specifically, we utilize foundation models such as DINOv2 [36] or CLIP [41] to extract visual features, which are then resized to the original image dimensions using interpolation. Next, we apply max pooling across features within regions defined by SAM-generated masks to obtain region-level features. These features are grouped into K clusters using KNN, categorizing them into distinct groups. Each region-level feature is assigned a group index, which we treat as pseudo-label for applying a class-level reweighted loss. We then count the number of regions in group as R2. This innovative approach allows us to effectively address the long-tail problem representation learning by balancing the influence of each group during the learning process. Hence, the dense distillation loss for the first R4 Rimiimin Where Ti = 1.0 - k; and W; -
k;
Mmax
stage is:
Laissill
M Σ wi (F2D, Fsp,i)
3.5 Maksed Token Prediction
As illustrated by VideoPrism [56], latent space reconstruction is an effective method for cross- modality knowledge distillation. Inspired by this, we propose two-stage framework to integrate latent space prediction within the MAE framework for 2D 3D knowledge distillation. This approach differs from previous 3D masked autoencoder methods like Point-MAE [37] and Bridge3D [10], which reconstruct raw, masked inputs as their targets.
As depicted in Fig. 2, our approach involves a two-stage process. In the first stage, we perform dense feature knowledge distillation from 2D to 3D using foundation models with the proposed SAM-guided tokenization method. In the second stage, we implement a teacher-student framework. The model from the first stage serves the teacher and is frozen for the second stage. During training, all tokens are processed by the teacher model to generate token features, and the student model is tasked with reconstructing these detailed 3D token representations using only the visible parts of the data. We introduce an instance- distillation loss to guide the student model's 's learning, pushing the limits of self- ceffsuupevviee learning in comprehending 3D spaces.
Specifically, we send complete point tokens to the teacher models and masked point tokens to the student models. For the instance-level knowledge prediction, pool all point token features after the teacher encoder as Fteacher and after the student encoder as Petudent The student model then ins
ins Fteacher using MLP layers. The instance prediction formulated as follows:
predicts ins
Line MSE(M F studentt ) Fins
ins pteacher))
(5)

---

SUN RGB-D
ScanNerV2
Methods
Pre-trained
AP25
Afso
2f
Al3o
VoteNet [39]
None
57.7
32.9
58.6
33.5
PointContrast [49]
57.5
34.5
59.2
38.0
Hou et al. [26]
36.4
39.3
4DContrast [9]
38.2
40.0
DepthContrast [55]
61.6
35.5
64.0
42.9
DPCo [30]
60.2
35.5
64.2
41.5
3DETR [35]
None None
58.0
30.3
62.1
37.9
+Plain Transformer
57.6
31.9
61.1
38.6
+Point-BBE BERT[S1]
61.0
38.3
+Point-MAB [37]
63.4
40.6
+MaskPoint [31]
63.4
40,6
+ACT [15]
63.5
41.0
+PiMAE [7]
59.9
33.7
63.0
40.2
+Bridge3D [10]
61.8
37.1
65.3
44.2
+Ours
63.5(41.7)
39.5(+2.4)
68.2 (42.9)
48.4(44.2)
GroupFree3D [33]
None
63.0
45.2
67.3
48.9
+Plain Transformer
None
62.2
45.0
66.1
48.3
+Point-MAB (37)
63.9
46.1
67 .4
49.8
+PiMAE [7]
65.0
46.8
67.9
50.5
+Bridge3D [10]
67.9
48.5
69.1
51.9
+Ours
68.9(+1.0)
52.1(+3.6)
72.3(+3.2)
55.7(+3.8)
Table 3D object detection results on ScanNet and SUN RGB-D dataset. We adopt the average precision with 3D IoU thresholds of 0.25 (AP25) and 0.5 (APto) for the evaluation metrics.
We use the global features of the student model with only visible inputs to predict the global features of the teacher model with complete inputs. Additionally, we employ a token-level prediction loss to ensure that the student models can predict the masked tokens obtained from the teacher model's decoder.
No
1
Ltoken
MSE(F Estudent Fleacher).
(6)
Nm
Where Nm is the number of masked tokens. The effectiveness this learning setup is evaluated through a defined reconstruction loss, ensuring high precision in the alignment between the student's s and teacher's outputs. Notably, we continue to employ the SAM-guided tokenization method to facilitate this process. The final loss for the second stage is formulated as:
C finnll Line Ltoken
4 Experiments
This section begins with an overview of the pre-training and fine-tuning configurations for our method. Subsequently, we demonstrate the method's effectiveness through its application to several prominent downstream tasks, such as 3D object detection and 3D semantic segmentation. Finally, we present comprchensive ablation studies to validate the impact and contribution of each component within our approach.
4.1 Self-supervised Pre-training and Fine-tuning
Pre-training, In our pre-training stage, we leverage the ScanNet dataset [14], aligning with ap- proaches adopted in prior research [39, 55] to obtain corresponding image and point cloud pairs. ScanNet is indoor dataset that includes approximately 1,500 scans derived from 2.5 million RGB-D frames. We follow the official protocol for training/validationn splits, extracting 78,000 frames from the training subset by sampling frame every 25 frames to construct dataset. For the
7

---

S3DIS
ScanNetV2
Methods
Pre-trained
mloU
mAcc
mloU
mAcc
SR-UNct [49]
None
68.2
75.5
72.1
80.7
PointContrast [49]
70.9
77.0
74.1
81.6
DepthContrast [55]
70.6
73.1
Hou et al. [26]
72.2
73.8
Standard Transformer [51]
None
60.0
68.6
PointBert [51]
60.8
69.9
PVIT [40]
None /
69.9
PVIT+Pix4Point [40]
69.6
75.2
Plain Transformer
None
61.1
67.2
67.3
73.1
+Point-MAE [37]
64.8
70.2
+Bridge3D [10]
70.2
76.1
73.9
80.2
+Ours
71.8 (+1.6)
78.2(+2.1)
75.4(+1.5)
81.5(+1.3)
Table 2: 3D semantic segmentation results S3DIS and ScanNet dataset. We adopt the mean accuracy (mAcc) and mean loU (mloU) for the evaluation metrics.
optimization process, we utilize the AdamW optimizer [34] throughout both stages of our training, starting with base learning rate of 0.001 and a weight decay set at 0.05. Our data is processed in batches 64. During the second stage of training, we increase the masking ratio (r.:) to 60%. To further enhance the training dynamics, we implement cosine learning rate scheduler coupled with drop path rate of 0.1 and include a warm-up phase of 10 epochs to facilitate a smooth adjustment the training conditions. For the 3D backbone encoder, we adopt the plain transformer structure used in Bridge3D [10]. On the image processing side, we employ the DINOV2 ViT-B model [36] to extract features. To adapt these features back to the original input size, we apply interpolation-bant up-sampling techniques. The training is conducted using four A100 GPUs.
Fine-tuning, Following Bridge3D [10], we remove the decoders used in pre-training and introduce task-specific decoders for various downstream tasks. A key distinction between fine-tuning approach and Bridge3D the use of SAM-guided tokenization to generate tokens, rather than the traditional KNN-based tokenization methods employed by Bridge3D. Additionally, for detection tasks, we do not introduce new query embeddings. Instead, we use the tokens generated through SAM- guided tokenization queries for self-attention. These tokens represent features of homogeneous neighboring regions defined by precise boundary regularities from SAM, making them suitable for 3D object detection tasks. Apart from these adjustments, adhere the same fine-tuning settings as Bridge3I for downstream tasks.
4.2 Results on Downstream Tasks
Object Detection. We demonstrate the generality of our proposed method by conducting pre- training on the indoor ScanNerV2 dataset [14] and subsequently fine-tuning it for object detection tasks in both the ScanNetV2 and SUN-RGBI [58] datasets. Building upon the baseline detection methods 3DETR [35] and GroupFrec3D [33], our method significantly outperforms the previous state-of-the-an method Bridge3D. Specifically, our performance surpasses Bridge3D by 2.9 and 4.2 in AP25 and AP using the 3DETR baseline, and by 3.2 and 3.8 in AP25 and AP using the GroupFree3D baseline on the ScanNetV2 dataset. This consistent improvement over Bridge3D underscores the efficacy of method in learning advanced 3D representations for object detection, indicating its potential for enhancing 3D scene understanding tasks.
Semantic Segmentation. In Table 2, we present the semantic segmentation results on the S3DIS [4] and ScanNet [14] datasets. Although Bridge3D [10] has improved the plain transformer baseline by large margin, our method still outperforms Bridge3D by 1.6 and 1.5 mIoU on the ScanNet and S3DIS datasets, respectively. It should be noted that Bridge3D utilizes foundation models with both 2D and text modalities, incorporating complex architectures.
contrast, our method, which utilizes only 2D foundation models, still outperforms Bridge3D in 3D semantic segmentation tasks. This demonstrates the efficiency of our proposed knowledge distillation strategy for enhancing 3D representation learning for semantic segmentation.

---

Dense Distillation
Masked Token Prediction
Balanced Re-weight
SAM-Guided Tokenzie
ScanNetV2
S3DIS
Af's
AP'o
mloU
mAce
61.1
38.6
61.1
67.2
62.4
41.7
66.2
71.3
64.5
44.3
68.7
74.1
66.0
69.7
75.9
67.1
47.0
70,9
77.0
68.2
48.4
71.8
78.2
Table 3: The effectiveness of each component. Ablation study on the effectiveness of each component on 3D object detection and semantic segmentation tasks.
ScanNetV2
S3DIS
AP26
APso
mJoU
m.
Stage Stage + MTP same stage
65.2
45.1
69.1
75.3
66.0
69.9
76.1
Stage + Stage 2 (Ours)
68.2
48.4
71.8
78.2
Table 4: The effectiveness of Stage. Ablation study the effectiveness of two-stage framework
on 3D object detection and semantic segmentation tasks. MTP here represents the masked token prediction
4.3 Ablation Study
The Effectiveness of Each Component. As illustrated in Table 3, the results effectively demon- strate the advantages of each component incorporated into our comprehensive framework. The detailed ablation study reveals that incorporating dense distillation significantly enhances 3D rep- resentation learning and improves overall system performance. Additionally, the implementation of a two-stage masked token prediction enables student models to learn well-aligned and highly contextualized representations across different modalities, thereby further enhancing overall system performance. Moreover, the introduction of balanced re-weighting mechanisms significantly boosts network performance by effectively mitigating the long-tail distribution challenge, which is inherently problematic in 3D datasets. Finally, the integration of SAM-guided tokenization marks the most substantial improvement within our framework, as it seamlessly aligns 2D and 3D features, thus avoiding potential conflicts and discrepancies in information transfer. In conclusion, each component of our proposed method designed to complementary to the others; their combined application not only achieves optimal results but also markedly enhances performance across both 3D object detection and semantic segmentation tasks, demonstrating the robustness and efficacy of our approach.
The Effectiveness of Each Stage. Table 4 underscores the effectiveness of our proposed two-stage method, which initiates with the distillation of dense representations from 2D foundation models into 3D models during the first stage. This initial phase of our study is meticulously designed to assess whether it is imperative to employ a teacher model if the strategy of predicting masked 2D features directly within the first stage could potentially achieve superior or equivalent performance. The results derived from this meticulous ablation study suggest that merely combining the initial stage with masked token prediction yields only modest improvements in overall performance. However, a significant enhancement is observed with the addition of the second stage, which more thoroughly integrates our structured teacher-student framework into the process. This marked improvement dis- tinetly highlights the critical importance of the teacher-student design within our innovative approach, confirming that the detailed and layered integration of these essential elements is absolutely vital for obtaining well-learned, robust representations that are crucial for effective 3D scene understanding tasks.
4.4 Apply on SOTA 3D Detectors
We recognize that applying method to state-of-the-art detection models can further demonstrate its generality and robustness. Therefore, we applied our approach to two leading 3D detection methods: CAGroup3D [47] and VDETR [45]. Due to the specifically designed BiResNet backbone

---

Methods
Pre-trained
ScanNet (AP25)
ScanNet (AP)))
CAGroup3D [4]
None
75.1
61.3
Ours
76.5
62.4
VDETR [5]
None
73.6
60.1
+ Ours
75.8
63.0
Table 5: 3D object detection results on ScanNet dataset based on CAGroup3D and VDETR.
Methods
ScanNef (AP25)
ScanNet (AP)))
ScanNet (mloU)
Scratch
61.1
38.6
67.3
CLIP2Scene [8]
62.0
40.1
69.2
Seal [32]
62.7
41.3
70.3
PPT [48]
62.8
42.1
70.9
Ours
68.2
48.4
75.4
Table 6: Comparison with other pre-training methods with different backbones on ScanNet dataset in 3D detection and semantic segmentation tasks.
used by CAGroup3D, we were able to apply only our SAM-guided knowledge distillation and representation re-weighting techniques
For VDETR, which reports results using both a modified ResNet34 encoder and a plain transformer encoder, we replaced its encoder with our pre-trained encoder and compared the performance the transformer backbone results reported in the original paper. The experimental results presented in Table. 5 show that our pre-training strategy enhances the performance of these state-of-the-an 3D detection models. Moreover, the performance improvement in VDETR, facilitated by our proposed AAM-guided tokenization and two-stage masked autoencoder, greater than that observed in CAGroup3D, highlighting the effectiveness of our approach.
4.5 Results Comparison with Pre- Pre-Training Methods for Other Backbones
In the main paper, we did not compare our results with Seal [32], PPT [48], and CLIP2Scene [8] as they use 3D-UNet the backbone and are exclusively fine-tuned for 3D semantic segmentation tasks. Most previous methods operate at the object-level or scene-level using transformer-based 3D models. To demonstrate the effectiveness of our approach specifically designed for transformers, we adapted the methodologies of Seal, PPT, and CLIP2Scene to the transformer structure, applying the same experimental settings as our method.
As shown in Table. 6, our method achieves the best performance, highlighting the advantages of our proposed strategies. In the revised version, will cite these papers and include discussion of their methodologies and results. We acknowledge that including comparisons with methods using different backbones could better illustrate the effectiveness of our approach, and therefore, have undertaken this additional evaluation. The results clearly demonstrate that our approach outperforms these existing methods, emphasizing the robustness and generalizability of our pre-training strategy.
5 Conclusion
In conclusion, our study addresses the challenges of aligning 2D and 3D representations and enhances knowledge distillation in self-supervised learning for 3D scene understanding. We introduced a novel SAM-guided tokenization method that aligns 3D transformer structures with region-level insights, improving distillation effectiveness. Additionally, we propose group-balke balaccd re- re-weighting strategy address the long-tail problem. Furthermore, we introduce a two-stage masked token prediction framework, enabling the student model to predict both global instance-level embeddings and local token-wise embeddings learned from the teacher model based on visible 3D input tokens. Experiments conducted on datasets such as SUN RGB-D, ScanNet, and S3DIS demonstrate the state-of-the-and performance of the proposed method in 3D object detection and semantic segmentation tasks. Our work is expected to have negative socictal implications.
10

---

References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning rep- resentations and generative models for 3d point clouds. arXiv preprint
2017.
[2] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Putlern Recognition, pages 9902-9912, 2022.
[3] Shaden Alshammari, Yu- Xioon Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition
via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6897-6907, 2022.
[4] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534-1543, 2016.
[5] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 156 1061915529, 2023.
[6] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint arXiv:2 022544 2021.
[7] Anthony Chen, Kevin Zhang. Renrui Zhang. Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. arXiv preprint arXivv:2303..mmmm 08 29, 2023.
[8] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou,
Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.049:
2023.
[9] Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Parl XXXII, pages 543-560. Springer, 2022.
[10] Zhimin Chen, Longlong Jing, Yingwei Li, and Bing Li. Bridging the domain gap: Self- supervised 3d scene understanding with foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[1]] Zhimin Chen, Longlong Jing, Liang Yang, Yingwei Li, and Bing Li. Class-level confidence based 3d semi-supervised learning. Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 633-642, 2023.
[12] Zhimin Chen, Yingwei Li, Longlong Jing, Liang Yang, and Bing Li. Point cloud self-supervised learning via 3d to multi-view masked autoencoder. arXiv preprint arXiv:2311. 10887, 2023. [13] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages
2019.
[14] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthis Nießner. Scannet: annotttad 3d reconstructions of indoor scenes. In Proceedings of the
2017. [15] Rumpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers
IEEE conference computer vision and pattern recognition, pages
help 3d representation learning? arXiv preprint arXiv:2212.085 2.03220, 2022.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 19929, 2020.
[17] Bi'an Du Xiang Gao, Wei Hu, and Xin Li. Self-contrastive learning with hard negative sampling for self-supervised point cloud learning. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3133-3142, 3142, 2021.
[18] Ziyue Feng, Longlong Jing, Peng Yin, Yingli Tian, and Bing Li. Advancing self-supervised monocular depth learning with sparse lidar. In Conference on Robot Learning, pages 685-694.
PMLR, 2022.
[19] Ziyue Feng, Liang Yang, Pengsheng Guo, and Bing Li. Cvrecon: Rethinking 3d geometric feature learning for neural reconstruction. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 17750-17760, 2023.
11

---

[20]
Ziyue Feng, Liang Yang, Longlong Jing. Haiyan Wang. YingLi Tian, and Bing Li. Disentangling object motion and occlusion for unsupervised multi-frame monocular depth. In European Conference Computer Vision, pages 228-244, Springer, 2022.
[21]
Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 103-118, 2018.
[22]
Kaveh Hassani and Mike Haley. Unsupervised multi-task feature learning on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8160-8171,
2019.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6000-16009, 2022.
[24] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303. 1113, 2023. [25] Georg Hess, Adam Tonderski, Christoffer Petersson, Lennart Svensson, and Kalle Aström. Lidarclip or: How i learned to talk to point clouds. arXiv preprint arXiv:22 16.5888, 2022. [26] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15587-15597, 2021.
[27] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self- self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6535-6545, 2021.
[28] Longlong Jing. Yucheng Chen, Ling Zhang, Mingyi He, and Yingli Tian. Self-supervised modal and view invariant feature learning. arXiv preprini arXiv:2005. 169 2020.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.026453 2023.
[30] Lanxiao Li and Michael Heizmann. A closer look at invariances in self-supervised pre-training for 3d vision. In Computer Vision-FiCCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX. pages 656-673. Springer, 2022.
[31] Haotian Liu, Mu Cai,, and Yong Jae Lee. Masked discrimination for sepf-supe learning on point clouds. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23, 2022, Proceedings, Part II, pages 657-675. Springer, 2022.
[32] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[33] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2949-2958, 2021.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1711:000 2017.
[35] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906-2917, 20222.
[36] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
[37] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I1, pages
604-621. Springer, 2022.
[38] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211. 15654, 2022.
[39] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference Computer Vision, pages 9277-9286, 2019.
[40] Guocheng Qian, Xingdi Zhang, Abdullah Hamdi, and Bernard Ghanem. Pix4point: Image pretrained transformers for 3d point cloud understanding. arXiv preprint arXiv:2208. 12259. 2022.
12

---

[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[42] Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In European Conference on Computer Vision, pages
626-642. Springer, 2020.
[43] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by recon- structing space. Advances in Neural Information Processing Systems, 32, 2019.
[44] Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar super-uue distillation for autonomous driving data. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9891-9900 2022.
[45] Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Ze Liu, Chunyu Wang, Han Hu, Nanning Zheng, and Baining Guo. V-detr: Detr with vertex relative position encoding for 3d object
detection. arXiv preprint arXiv:2308. 04409, 2023.
[46] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[47] Haiyang Wang. Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang. et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. Advances in Neural Information Processing Systems, 35:299755
2022.
[48] Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng. Xihui Liu, Kaicheng Yu, and Hengshuang Zhao. Towards large-scale 3d representation learning with multi-dataset point prompt training. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition, pages 19551-19562, 2024.
[49] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European conference on computer vision, pages 574-591. Springer. 2020.
[50] Sihao Yu, Jiafeng Guo, Ruqing Zhang. Yixing Fan, Zizhen Wang, and Xueqi Cheng- A re-balancing strategy for class-imbalanced classification based on instance difficulty. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
70-79, 2022.
[51] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert:
Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the Computer Vision and Pattern Recognition, pages 19313-19322, 2022. [52] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.0477
IEEE/CVF Conference
2023.
[53] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances in neural information processing systems. 35:27061
2022.
[54] Renrui Zhang, Ziyu Guo, Wei Zhang. Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1552-8562, 2022.
[55] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of
3d features on any point-cloud. In P Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10252-10263. 2021.
[56) Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, al. Videoprism: A foundational visual encoder for video understanding, arXiv preprint arXiv:2402.1322.
2024.
[57] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
1009-1018, 2019.
[58] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
[59] Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:2112.01071, 2021.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: in Abstract and Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: in Limitations and Future Work
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Disclose in Self-supervised Pre-training and Fine-tuning
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
Code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines
ce/
public/ /gwi/essssse//eeeee Subdmu ss ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht (https: / / //nips.... cc http/publicct http//uisssss./eee Code Submi as i ionooooiii icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: Specify in Self-supervised Pre-training and Fine-tuning
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in Self-supervised Pre-training and Fine-tuning.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips...... wwtpppblccc/tccccssulceeeeeeeeee
Answer: [Yes]
Justification: Readed and conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Experiments done with public dataset and open-source foundation models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Have no risk
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Cited all relative works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thcode. com da t t has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assests are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "Acrpd0c686", "reviewer": "Reviewer_KyqU", "summary": "The paper proposes a self-supervised method for understanding 3D scenes by predicting the 3D mask of the point cloud. The masks are initialized from Segment Anything (SAM), followed by two stages of the knowledge distillation framework to train the 3D teacher and student networks. The method is evaluated on SUN RGB-D, ScanNet, and S3DIS datasets.", "questions": "refer to the weaknesses", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is generally well-written and easy to follow.", "weaknesses": "- The first concern is the method's innovation. Other methods, like Seal [P1], have already proposed the idea of distilling the knowledge from 2D foundation models into the 3D network for mask prediction. However, it is not discussed or compared.\n\n- The insight into method design is not elaborated. It is not clear why two-stage training is necessary for self-supervised learning. How about three-stage, four-stage, or momentum updating proposed in MeanTeacher? It is better to present more insight and intuitive explanations. Besides, it is difficult to understand why group-balanced re-weighting will work since the pseudo labels are also extremely noisy, i.e., it also suffers from long-tail issues.\n- More recent methods should be comparied to verify the efficientness of the proposed method. For example, CLIP2Scene is disscussed but not comparied. Seal [P1], also utilize 2D models mask prediction to regularize 3D network.\n\nP1. Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.\n\n\n\n**Q1, innovation of method.**\n\nPlease refer to the first part of the general response at the beginning.\n\n\n**Q2, the insight of the two-stage design.**\n\n\nMasked token prediction has proven effective for single-modality learning [4]. Our method extends this concept to cross-modality knowledge distillation by designing an efficient two-stage framework for cross-modality masked feature prediction. Our two-stage design works as follows: *In the first stage*, we perform knowledge distillation using SAM-guided tokenization and SAM masks to seamlessly transfer 2D region-level information to 3D. *In the second stage*, we use masked-view inputs to predict contextualized 3D representations within a latent space aligned by a teacher model with complete-view inputs. *This ensures that student models learn well-aligned and contextualized representations*, leveraging the proven efficiency of masked feature prediction for multi-modality self-supervised learning.\n\nThe ablation study presented in Table 5 of the main paper (also shown in Table 1 below) demonstrates that the two-stage design outperforms the one-stage. In the one-stage design, the 3D encoder directly predicts masked parts of the 2D features from foundation models, resulting in poorer performance. This underperformance is likely due to the significant domain gap between 2D and 3D models, which causes the network to learn sub-optimal features when applying one-stage masked 2D feature prediction.\n\nOur two-stage design addresses this issue by separating the process into two distinct phases: the first stage focuses on 2D to 3D knowledge distillation, and the second stage handles 3D masked token prediction. This approach effectively reduces the domain gap between 2D and 3D representations, leading to better representations and improved performance.\n\n**Q3, reasons for two stage design instead, three, or even four stages.**\n\nIn the second stage, the teacher model is frozen, and the student model is trained to predict the masked feature tokens. Since the teacher model's weights remain unchanged, adding more stages would only extend the training duration for the student encoder in the second stage without providing additional benefits. Therefore, adding additional stages to train the student encoder, such as a third or fourth, is unnecessary. The experiments shown in Table 1 below indicate that introducing a third or fourth stage yields similar results to the two-stage setting even with more training time.\n\n| Stage settings      | ScanNet ($AP_{25}$) | ScanNet ($AP_{50}$) |\n|---------------------|-----------------------|-----------------------|\n| One-stage setting   | 66.0                  | 46.3                  |\n| Two-stage setting   | 68.2                  | **48.4**              |\n| Three-stage setting | 67.7                  | 48.3                  |\n| Four-stage setting  | **68.4**              | 47.5                  |\n\n**Table 1:** Ablation study for stage settings.\n\n\n\n\n\n**Q4, why not utilize momentum updating?**\n\nWhen utilizing momentum updating, two augmented positive views are sent to both the trainable encoder and the momentum encoder to obtain positive representation pairs or pseudo-label pairs. The weights of the momentum encoder are then updated with the weight of the trainable encoder through the contrastive loss or the supervision loss via pseudo-labels. However, our method does not require two augmented views for training in either stage; therefore, momentum updating is not applicable to our approach.\n\n\n**Q5, why group-balanced re-weighting will work.**\n\nGroup-balanced re-weighting is introduced to address the long-tail problem inherent in the natural imbalance of object class occurrences. As discussed in the paper [5], *foundation models provide well-represented patch-level features that can effectively identify different classes through clustering-based methods due to their training on large datasets, making it effective as a zero-shot learner.*\n\n\nIn this paper, we take advantage of the foundational model to extract object-level features and apply a clustering method to compute distribution statistics. These statistics are less noise-prone because the foundational model provides robust feature representation for both head and tail class objects. We then normalize the distribution factors as weights to regularize the distillation process, offering an effective solution to the long-tail issue.\n\n\n**Q6, comparison with CLIP2Scene and Seal.**\n\nPlease refer to the second part of the general response at the beginning.\n_________________\n\n\n\n[1] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023\n\n[2] Bridging the domain gap: Self-supervised 3d scene understanding with foundation models, NeurIPS 2023\n\n[3] Videoprism: A foundational visual encoder for video understanding, ICML 2024\n\n[4] Self-supervised learning from images with a joint-embedding predictive architecture, CVPR 2023\n\n[5] Deep ViT Features as Dense Visual Descriptors, ECCVW 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ouFvpAWy9v", "reviewer": "Reviewer_EQZo", "summary": "The paper proposes a 3D transformer tokenization technique to align 3D representations with 2D ones, distilling from 2D pre-trained knowledge from SAM. The method achieves favorable performance on 3D object detection and semantic segmentation compared to prior self-supervised learning methods.", "questions": "* What's the input to $F_{2D, i}$ and $F_{3D, i}$ in Eq (1)? Where is $i$ sampled from? \n* $1/O_i$ in Eq (1) should probably be written as $1/|O_i|$.\n* What is $n_{min}$ and $n_{max}$ in line 209?", "limitations": "Limitations are discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper provides a good explanation contrasting the proposed method with prior works that develop 3D representations distilling from 2D foundation models, highlighting the drawbacks of KNN-based point tokenization which well motivates the proposed solution. \n* Several techniques, such as group-balanced re-weighting and two-stage teacher-forcing training, are adopted in the framework and ablations show their effectiveness.\n* The performance is strong, with advantages compared to prior self-supervised methods.", "weaknesses": "* It is not discussed why SAM is chosen as the distillation source for tokenization, instead of other 2D foundation models such as DINO, or even generative ones such as Stable Diffusion.\n* The representation is evaluated on the task of 3D object detection and 3D semantic segmentation, which intuitively seem to correspond well to the strength of SAM as SAM may have implicitly learned the notion of objects and object semantics during its training phase. Are there tasks where the proposed representation could be useful?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and insightful comments! We have addressed your questions as follows.\n\n\n**Q1, why SAM is chosen as the distillation source for tokenization**: \n\nWe chose SAM [1] to guide tokenization because it generates great zero-shot masks that provide boundary regularities and effectively facilitate region-level knowledge distillation. In contrast, other foundational models, such as Stable Diffusion [2] or DINOv2 [3], cannot generate comparable zero-shot masks.\n\nFor feature distillation, we use DINOv2 as the teacher model, enabling the student 3D models to predict the same region-level features obtained from DINOv2. We conducted an ablation study comparing features obtained from other foundational models like SAM, Stable Diffusion, and CLIP [4] to extract visual features. The results in Table 1 below indicate that utilizing DINOv2 achieves the best performance.\n\n\n| Foundation Models | ScanNet ($AP_{25}$)  | ScanNet($AP_{50}$)  |\n|-------------------|---------------------|---------------------|\n| Stable Diffusion [2] | 66.2                | 45.7               |\n| CLIP [4]          | 66.8               | 46.3                |\n| SAM [1]           | 67.5                | 46.7                |\n| DINOv2 [3]        | **68.2**            | **48.4**            |\n\n**Table 1:** Ablation study for the choice of foundation models for representation distillation.\n\n\n\n\n\n**Q2: other tasks where the proposed representation could be useful.**\n\n\nReviewer icGN suggests that our method could also be applied in online 3D perception settings. To demonstrate the generality of our approach, we applied it to the online 3D perception method described in [5]. However, due to differences in the backbone architecture, our pre-trained encoder cannot be directly integrated with the 3D detection method FCAF3D [6] used in [5]. Therefore, we replaced FCAF3D with the transformer-based VDETR [7] for online detection. The experimental results in Table 2 indicate that our method enhances performance in the online 3D perception setting, demonstrating the generality and effectiveness of the proposed pre-training approach.\n\nAdditionally, by replacing the DINOv2 teacher model with CLIP, our method could potentially enable 3D zero-shot semantic segmentation by calculating the feature similarity between the predicted features and the CLIP text features. Furthermore, since our method distills features from 2D foundation models into 3D networks, it can enhance various 3D tasks that benefit from 2D features, such as 3D scene classification, 3D reconstruction, and 3D instance segmentation. However, due to time limitations, we were unable to apply our method to these tasks in the current study, but we plan to do so in the future to further demonstrate the generality of the proposed method.\n\n| Methods    | Pre-trained | ScanNet ($AP_{25}$) | ScanNet ($AP_{50}$) |\n|------------|-------------|-----------------------|-----------------------|\n| VDETR-online  [5]   | *None*           | 68.9       | 52.7      |\n| VDETR-online + ours      | *$\\checkmark$*       | **71.3**                  | **55.8**                |\n\n**Table 2:** Online 3D object detection results on ScanNet dataset.\n\n\n**Q3, some definitions.**\n\n**What is the input to the $F_{2D,i}$ and $F_{3D,i}$?**\n\nAs illustrated in the main paper (lines 174 to 177), $I$ represents pixel-level features obtained from raw images processed by foundation models and feature interpolation. $H_i$ represents the SAM-guided tokenized features obtained from the raw point clouds with the $i$-th masks generated by SAM. In Equation 1, we utilize the $i$-th masks to group the pixel-level image feature representations $I$ into the $i$-th object-level features to obtain $F_{2D,i}$ and utilize one MLP project layer to project $H_i$ to the $F_{3D,i}$. Therefore, the inputs for $F_{2D,i}$ and $F_{3D,i}$ are the raw images, point clouds, and the $i$-th masks generated by SAM.\n\n**Where is the i sampled from?**\n\n$i$ represents the $i$-th masks generated by SAM.\n\n**What is $n_{min}$ and $n_{max}$?**\n\n$n_{min}$ and $n_{max}$ represent the minimum and maximum numbers of feature clusters.\n\n**$\\frac{1}{O_i}$ in Eq (1) should probably be written as $ \\frac{1}{|O_i|}$**\n\nThanks for pointing out! We will correct $\\frac{1}{O_i}$ in equation (1) to $ \\frac{1}{|O_i|}$.\n\n_________________\n\n[1] Segment Anything, CVPR 2023\n\n[2] High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022\n\n[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024\n\n[4] Learning Transferable Visual Models From Natural Language Supervision, ICML, 2021\n\n[5] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024\n\n[6] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection, ECCV 2022\n\n[7] V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HMsE6RcBQe", "reviewer": "Reviewer_DTv4", "summary": "This paper introduces a novel method to enhance 3D scene understanding by addressing misalignment between 2D and 3D representations and the long-tail distribution in 3D datasets. The proposed approach involves a SAM-guided tokenization method for seamless alignment and a group-balanced re-weighting strategy to handle representation imbalance. It also incorporates a two-stage masked token prediction process for effective knowledge distillation from 2D foundation models to 3D networks.", "questions": "What is training time for fine-tuning? How about memory consumption compared to Bridge3D? \n\nBridge3D already mentioned in their paper that one of their limitations is that the current work primarily focuses on indoor 3D scene understanding. Is the same limitation applied for this paper?", "limitations": "Can this method apply to outdoor scene understanding?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The SAM-guided tokenization method effectively aligns 2D and 3D representations, overcoming the limitations of traditional KNN-based tokenization techniques.\n\nThe introduction of a group-balanced re-weighting strategy addresses the long-tail distribution problem in 3D datasets, improving the representation of under-represented samples.", "weaknesses": "Mask generation is an important step of the proposed method, probably a visual comparison with current SOTA methods would be more helpful to justify its effectiveness.\n\nThere are typos in this manuscript, and the reference style is very confused (e.g., Bridge3D (10)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and insightful comments! We have addressed your questions as follows.\n\n**Q1, visual comparison with other mask generation methods.**\n\nWe have added both visual and results comparisons with other mask generation strategies, such as superpoint and superpixel, to justify the effectiveness of SAM. Visualizations of these comparisons are included in the attached PDF file. Table 1 below indicates that utilizing SAM for mask generation achieves the best performance.\n\n\n| Masking strategy     | ScanNet ($AP_{25}$) | ScanNet ($AP_{50}$) |\n|---------------------|-----------------------|-----------------------|\n| Superpixel   | 66.0                 | 45.7                  |\n| Superpoint   | 66.8                  | 46.1              |\n| SAM | **68.2**                  | **48.4**                  |\n\n**Table 1:** Ablation study for different mask generation methods including superpixel, superpoint, and SAM.\n\n\n\n\n\n**Q2, what is training time for fine-tuning? How about memory consumption compared to Bridge3D?**\n\nWe have included the training times in Table 2 below. Training Bridge3D requires approximately 31GB of memory, our method requires a total of 36GB of memory. \n\n| Methods       | SUN RGB-D | ScanNet |\n|---------------|-----------|-----------|\n| 3DETR         | 29 hours  | 11 hours  |\n| GroupFree3D   | 15 hours  | 4 hours   |\n\n**Table 2:** Fine-tuning times for 3DETR and GroupFree3D on SUN RGB-D and ScanNetV2 datasets with 2 A100 GPUs.\n\n\n**Q3, application in the outdoor scene understanding.**\n\nOne key motivation for our method is to address the 2D-3D alignment challenges posed by traditional KNN-based tokenization methods. In outdoor scenarios, point clouds are often very sparse, especially at long-range distances, making KNN-based tokenization methods unsuitable. Current transformer-based methods for outdoor environments utilize voxel or pillar methods for feature extraction, treating these voxels or pillars as 2D images. Due to significant differences between outdoor and indoor environments and the sparsity of point clouds, our SAM-guided tokenization may not be suitable for outdoor scenarios. However, other parts of our method, such as representation re-weighting and two-stage masked token prediction, can be effectively applied in outdoor contexts.\n\nWe have chosen BEV-MAE [2] as our baseline method to distill 2D features into 3D BEV maps using the proposed re-weighting strategy. In the second stage, we enable the student network to reconstruct the masked BEV features obtained from the first-stage teacher models. As shown in Table 3 below, our re-weighting strategy and two-stage masked token prediction also benefit outdoor scenarios. Due to time constraints, we were unable to fully optimize the training parameters, indicating that there is still room for performance improvement.\n\n\n\n| Methods    | Waymo ($mAP$) | Waymo ($APH$) |\n|------------|---------------|---------------|\n| Scratch    | 65.60         | 63.21         |\n| GD-MAE [1] | 66.9          | 64.53         |\n| BEV-MAE [2]| 67.02         | 64.55         |\n| Ours       | **67.81**     | **64.93**     |\n\n**Table 3:** 3D detection results in the outdoor Waymo dataset.\n\n**W1, typos, and format issues.**\n\nThanks for pointing out! We will correct the typos and update the reference style in the revised version of the paper.\n\n_________________\n\n\n[1] GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds, CVPR 2023\n\n[2] BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios, AAAI 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMz0KIsawz", "reviewer": "Reviewer_icGN", "summary": "This paper proposes a two-stage SAM-guided pre-training method for 3D scene understanding. The authors present a group-balanced re-weighting method for long-tail representation distillation and a SAM-guided tokenization method to seamlessly align 2D and 3D region-level features. Extensive experiments on various downstream tasks show the effectiveness of the proposed pre-training method.", "questions": "It seems the models are pretrained on independent RGB-D frames, and then finetuned on reconstructed room-level point clouds. There may be some domain gap between the single-view point clouds and the complete room-level ones. I wonder is this work applicable to the online 3D perception setting [1, 2], which is more relevant to RGB-D perception and is a more valuable setting.\n\n[1] Fusion-aware point convolution for online semantic 3d scene segmentation, CVPR 2020\n[2] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024", "limitations": "Limitations is discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Adopt SAM to guide pre-training makes sense, since the semantic information can help model to learn high-quality patterns.\n2. The results are good, surpassing previous state-of-the-art methods significantly.", "weaknesses": "1. For 3D object detection results, even with the proposed pre-training method, the model's overall mAP is lower than SOTA 3D detector like CAGroup3D and VDETR. Can this work be applied on these more advanced architectures? If so, supporting experimental results are required.\n2. Using SAM to guide 3D pretrained has already been explored in SEAL [1], more comparison or discussion is required.\n3. This paper should not only compare with transformer-based pretraining methods, but also other state-of-the-art pretrianing techniques like PPT [2].\n\n\n\n[1] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023\n[2] Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and valuable suggestions. We have addressed your questions as follows.\n\n\n**Q1, apply on SOTA 3D detectors.** \n\nIn the main paper, we did not apply our method to other state-of-the-art detection methods because those works often incorporate specialized design structures different from the pre-training backbones, making comparisons less fair. *For pre-training tasks, comparisons are typically made with previous pre-training methods using the same structures. For example, Seal [1] only conducts training on the 3D U-Net and compares its results with previous outdoor pre-training methods.* The overall performance of Seal, as reported in the paper, is also lower than state-of-the-art 3D semantic segmentation methods like 2DPASS [2] and LidarMultiNet [3] with full supervision on the nuScenes dataset.\n\nHowever, we understand that applying our method to state-of-the-art detection methods can further demonstrate its generality. Therefore, we applied our approach to the state-of-the-art methods CAGroup3D [4] and  VDETR [5]. Since CAGroup3D utilizes a specifically designed BiResNet backbone, we could only apply our SAM-guided knowledge distillation and representation re-weighting method to it. For VDETR, which reports results with both a modified ResNet34 encoder and a plain transformer encoder, *we replaced its encoder with our pre-trained encoder and compared it to the **transformer backbone results reported in the original paper**.* The experiments in Table 1 indicate that our pre-training strategy enhances the performance of these state-of-the-art 3D detection methods. Furthermore, with the proposed SAM-guided tokenization and two-stage masked autoencoder, the performance improvement in VDETR is larger than in CAGroup3D, demonstrating the effectiveness of our method.\n\n\n\n\n\n| Methods    | Pre-trained | ScanNet ($AP_{25}$) | ScanNet ($AP_{50}$) |\n|------------|-------------|-----------------------|-----------------------|\n| CAGroup3D [4]  |  None     | 75.1                  | 61.3                  |\n| + Ours     | $\\checkmark$            | **76.5**       | **62.4**       |\n| VDETR [5]     | None     | 73.6                  | 60.1                  |\n| + Ours     | $\\checkmark$           | **75.8**       | **63.0**       |\n\n**Table 1:** 3D object detection results on ScanNet dataset based on CAGroup3D and VDETR.\n\n\n\n**Q2, comparison with Seal.**\n\nPlease refer to the first part of the general response at the beginning.\n\n\n\n**Q3, comparison with other structure pre-training techniques like PPT.**\n\nPlease refer to the second part of the general response at the beginning.\n\n\n**Q4, domain gap between the single-view point clouds and the complete room-level ones.**\n\nThis is a great question. Previous 3D scene understanding pre-training strategies like PointContrast [6], DepthContrast [7], and PiMAE [8] all pre-train on single-view point clouds before fine-tuning on complete room-level point clouds. As demonstrated in PointContrast, *direct pre-training on complete room-level point clouds yields poorer results compared to single-view point clouds.* This may be due to several factors, including a more abundant and diverse set of training samples and the regularization effect of natural noise from camera instability.\n\nThe limited domain gap between pre-training on single-view point clouds and fine-tuning on room-level multi-view point clouds can be attributed to several reasons. Pre-training on single-view point clouds helps the network learn detailed local features and geometric structures, which makes the network align well with the of room-level point clouds. This phase also enhances the network's robustness to various perspectives and partial visibility, facilitating effective knowledge transfer. \n\n**Q5, applicable to the online 3D perception setting.**\n\nOur method can also be applied to the online 3D perception setting. To demonstrate the generality of our approach, we applied it to the online 3D perception work [9]. However, due to differences in the backbone architecture, our pre-trained encoder cannot be directly integrated with the 3D detection method FCAF3D [10] used in [9]. Therefore, we replaced FCAF3D with VDETR [5] for the online detection. The experimental results in Table 2 below indicate that our method enhances performance in the online 3D perception setting, demonstrating the generality and effectiveness of the proposed pre-training approach.\n\n\n\n| Methods    | Pre-trained | ScanNet ($AP_{25}$) | ScanNet ($AP_{50}$) |\n|------------|-------------|-----------------------|-----------------------|\n| VDETR [5]  | *None*      | 73.6                  | 60.1                 |\n| VDETR-online  [9]   | *None*           | 68.9       | 52.7      |\n| VDETR-online + ours      | *$\\checkmark$*       | **71.3**                  | **55.8**                |\n\n**Table 2:** Online 3D object detection results on ScanNet dataset.\n\n_________________\n\n[1] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023\n\n[2] 2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds, ECCV 2022\n\n[3] LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception, AAAI 2023\n\n[4] CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds, NeurIPS 2022\n\n[5] V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection, ICLR 2024\n\n[6] PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding, ECCV 2020\n\n[7] Self-Supervised Pretraining of 3D Features on any Point-Cloud, ICCV 2021\n\n[8] PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection, CVPR 2023\n\n[9] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024\n\n[10] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection, ECCV 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**
--
We sincerely thank each reviewer for their thoughtful feedback and detailed reviews. We address the main concerns regarding novelty and comparisons with other peer research below.

**Comparison with Seal and the innovation of methodology.**

Thank you for highlighting Seal [1] as a related work. We did not discuss and compare it with our work because *Seal uses a different backbone (3D U-Net), is pre-trained on different datasets (outdoor scenes), and is fine-tuned only for 3D semantic segmentation tasks.* 3D U-Net struggles with scalability and flexibility, making it less effective for scaling and handling tasks such as detection. Furthermore, we have already conducted a detailed comparison with the most relevant work, Bridge3D [2], which employs a similar strategy to Seal’s by leveraging SAM masks and 2D features. Both Bridge3D and Seal directly use SAM masks to group 2D and 3D object-level features for object-level contrastive learning. We discuss the differences between our method and these SAM-guided group contrastive learning methods in lines 32 to 47 of the main paper. **Notably, our dense distillation setting in the ablation study shown in Table 3 of the main paper is the same as the Seal method**, except that Seal utilizes the InfoNCE loss, whereas dense distillation leverages the SmoothL1 loss. To further demonstrate the effectiveness of our method, we will cite and compare Seal in the revised version of our paper. The comparison results with Seal are included in Table 1 below.

Compared to Seal, our method is unique in *addressing two critical challenges*: the misalignment of object-level 2D and 3D features for  traditional KNN-based tokenization methods, and the imbalance problem of representations during the distillation. *To solve the first issue*, we propose a SAM-guided 3D tokenization method that seamlessly aligns 2D and 3D object-level features. *For the second challenge*, we introduce a self-supervised learning re-weighting strategy to enhance the weight of tail representations. *Furthermore, another novel aspect* of our approach is extending previous single modality masked token prediction into cross-modality learning with a two-stage framework design to learn well-aligned and contextualized 3D representations.


**Results comparison with pre-training methods for other backbones like Seal [1], PPT [3], and CLIP2Scene [4].**

In the main paper, we did not compare our results with Seal [1], PPT [3], and CLIP2Scene [4] as they utilize 3D-UNet as the backbone and are fine-tuned exclusively for 3D semantic segmentation tasks. Most previous object-level and scene-level 3D transformer-based pre-training methods, such as I2P-MAE [5], PiMAE [6], and Bridge3D [2], *do not directly compare their approaches with other pre-training strategies based on other 3D backbones like PointNet, DGCNN, or 3D-UNet.* Hence, we followed the same comparison strategy.

However, we recognize that including comparisons with methods using other backbones could better illustrate the effectiveness of our approach. Therefore, we applied the methodologies of Seal, PPT, and CLIP2Scene to the transformer structure and utilized the same settings as our method. As shown in Table 1 below, our method achieves the best performance, highlighting the advantages of our proposed strategies. In the revised version, we will cite those papers and include a discussion of their methodologies and results.


| Methods        | ScanNet ($AP_{25}$) | ScanNet ($AP_{50}$) | ScanNet ($mIoU$) |
|----------------|-----------------------|-----------------------|--------------------|
| Scratch        | 61.1                  | 38.6                  | 67.3               |
| CLIP2Scene [4] | 62.0                  | 40.1                  | 69.2               |
| Seal [1]       | 62.7                  | 41.3                  | 70.3               |
| PPT [3]        | 62.8                  | 42.1                  | 70.9               |
| Bridge3D [2]   | 65.3                  | 44.2                  | 73.9               |
| Ours           | **68.2**              | **48.4**              | **75.4**           |

**Table 1:** Comparison with other state-of-the-art pre-training methods on ScanNet dataset with 3D detection and semantic segmentation tasks.


The attached PDF contains a table and a figure, where the figure presents a visual comparison with other mask generation methods to address the question from Reviewer DTv4. Additionally, the table includes the ablation study results of different mask-generation strategies.

_________________


[1] Segment Any Point Cloud Sequences by Distilling Vision Foundation Models, NeurIPS 2023

[2] Bridging the domain gap: Self-supervised 3d scene understanding with foundation models, NeurIPS 2023

[3] Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training, CVPR 2024

[4] Towards Label-efficient 3D Scene Understanding by CLIP, CVPR 2023

[5] Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders, CVPR 2023

[6]PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection, CVPR 2023

Meta-review: The paper received three Borderline accepts and one Accept after the rebuttal and discussion period. All reviewers acknowledged the novelty and effectiveness of the proposed method, and the AC found no reasonable cause to overturn all of the reviewers' positive evaluations. Therefore, considering the submitted paper, the reviewers' comments, the authors' rebuttals, and the discussion, the AC finds it reasonable to conclude that the paper is acceptable. However, the AC recommends a clearer description of the novelty of the method and qualitative and quantitative comparisons with state-of-the-art methods such as Seal.
Decision: Accept (poster)