Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models
Authors: Zhimin Chen
Longlong Jing
Yingwei Li
Bing Li
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D self-supervised learning
Multi-modal Representation Learning
Masked autoencoders
Knowledge distillation

Abstract: Foundation models have achieved remarkable results in 2D and language tasks like image segmentation, object detection, and visual-language understanding. However, their potential to enrich 3D scene representation learning is largely untapped due to the existence of the domain gap. In this work, we propose an innovative methodology called Bridge3D to address this gap by pre-training 3D models using features, semantic masks, and captions sourced from foundation models. Specifically, our method employs semantic masks from foundation models to guide the masking and reconstruction process for the masked autoencoder, enabling more focused attention on foreground representations. Moreover, we bridge the 3D-text gap at the scene level using image captioning foundation models, thereby facilitating scene-level knowledge distillation. We further extend this bridging effort by introducing an innovative object-level knowledge distillation method that harnesses highly accurate object-level masks and semantic text data from foundation models. Our methodology significantly surpasses the performance of existing state-of-the-art methods in 3D object detection and semantic segmentation tasks. For instance, on the ScanNet dataset, Bridge3D improves the baseline by a notable margin of 6.3%. Code will be available at: https://github.com/Zhimin-C/Bridge3D

Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models
Zhimin Chen' Clemson University shiminc@clemson. edu
Longlong Jing? The City University of New York Ljingggradcenter cuny edu
Yingwei Li Johns Hopkins University yingwei. 1i@ jhu. edu
Bing L Clemson University b 01i48clemson. edu
Abstract
Foundation models have achieved remarkable results in 2D and language tasks like image segmentation, object detection, and visual-lagguage understanding. However, their potential to enrich 3D scene representation learning largely untapped due to the existence of the domain gap. In this work, we propose an innovative methodology called Bridge3D to address this gap by pre-training 3D models using features, semantic masks, and captions sourced from founda- tion models. Specifically, our method employs semantic masks from foundation models to guide the masking and reconstruction process for the masked autoen- coder, enabling more focused attention on foreground representations. Moreover, we bridge the 3D-text gap at the scene level using image captioning foundation models, thereby facilitating scene-level knowledge distillation. We further ex- tend this bridging effort by introducing innovative object-level knowledge distillation method that harnesses highly accurate object-level masks and seman- tic text data from foundation models. Our methodology significantly surpasses the performance of existing state-of-the-ar methods in 3D object detection and semantic segmentation tasks. For instance, the ScanNet dataset, Bridge3D improves the baseline by a notable margin of 6.3%. Code will be available at: nttps://githeen com Bttm//hhmmiCC/Biieme/33
Introduction
In recent years, task-agnostic pre-trained representations have fundamentally reshaped the landscape of Natural Language Processing (NLP), driven by the success of foundation models such as GPT- 3 [55], PALM [13], T-NLG [31], and BERT [7]. Parallel advancements have been observed in the realm of computer vision, where foundation models like CLIP [54], Grounding DINO [43], DINOV2 [49], BLIP [39], and SAM [37] have established new benchmarks in 2D vision tasks, emerging as the leading approach for achieving state-of-the-art performance. However, the potential of these powerful models in advancing 3D scene understanding is yet to be fully realized, primarily due to the limited availability of large-scale 3D-text pair datasets and the considerable cost associated with procuring high-quality 3D annotations. Despite recent studies demonstrating the potential of individual foundation models like CLIP [54] or MOCO [27] in enhancing 3D scene understanding, comprehensive exploration of the utility of other foundation models and their synergistic combinations remains a largely uncharted territory.
To address this challenge, propose novel framework Bridge3D that hamnesses the strengths of multiple foundation models to advance 3D representation learning through self-supervised learning
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Captioning
How can foundation models help 3D understanding?
A waiting room in building with walls, chair, table, desk.
2D Semantic
Features
Chair
Desk
The motivation of Bridge3D. The driving force behind Bridge3D is to create method that bridges the gap between 3D models and text/image foundation models via self-supervised learning.
Figure 1:
approach. Specifically, the Bridge3D leverages image captioning outputs from foundation models to generate 3D scene prompts, establishing bridge between 3D and text domains for scene-level knowledge distillation. These generated prompts further utilized produce instance segmentation results with the assistance of Grounding DINO and SAM. Subsequently, we employ 3D network trained self-supervised task that distillates the knowledge from text and 2D point clouds the scene level. This is followed by the distillation of these multidimensional features into 3D features at the object level. Additionally, to optimize point reconstruction, we propose an inventive masking and patch-dropping strategy that redirects the model's 's attention toward foreground object representation learning.
Our proposed framework effectively circumvents three significant hurdles in 3D self-supervised learning. Firstly, the adoption of foundation models in novel masking and dropping strategy allows the network to concentrate more on foreground object representation learning, thereby en- hancing the performance of the 3D model. This is distinct shift from traditional 3D masked autoencoder methods that rely on random masking strategy and reconstruct all point clouds, which impairs representation learning due the imbalance between foreground and background points. Secondly, the lack of datasets incorporating both 3D and text description pairs significantly hampers the potential for large-language models to contribute to 3D understanding. To overcome this obstacle, Our method first employs image captioning to generate text descriptions from paired images of point clouds, effectively bridging the 3D-text gap the scene level. This novel integration of 3DD and text modalities presents compelling new frontier for improving self- suffvuupe 3D scene understanding. Lastly, our approach stands in contrast to previous methodologies [9; 59] that facilitated either 2D to 3D or text to 3D distillation in isolation due to inherent limitations in mask generation. Instead, our strategy leverages foundation models to generate highly precise object-level masks and semantic text information. This approach seamlessly integrates object-level 3D, visual, and textual features, thereby significantly enhancing the quality 3D scene representation learning.
We evaluate our method on multiple datasets, including SUN RGB-D [75] and ScanNet [14] for 3D object detection and S3DIS [5] for 3D semantic segmentation. Our approach outperforms state-of-the-an self self-supervised learning methods in both tasks, demonstrating the effectiveness of our proposed framework. The contributions of Bridge3D can be summarized as follows:
1. We propose novel masking and patch-dropping strategy based on foundation models to refine the focus the network on foreground representation learning for 3D masked autoencoders.
2. We propose novel scene-level and object-level multi-modality knowledge distillation method that pre-trains 3D network via features, semantic masks, and captions obtained from foundation models

---

Scene level
Text-D Disill
Semeelevd
20-30 Disill
Object-Level
2D-3D Distill
Object-Lon evdl Text-3D Diviill
Head
Pooling
Mask Cantted Pool
Global Pool
Mask Guided
Pool
Scone Frature
Object
Head
Howl
3D Encoder
Viesal Encoder
Tixt Fraardkr
Resorstuction Hoal
wailing building with walls, thair, table desk
Semantic
Gulded
Drop
Semaniic Guided
Masking
Figure 2: Overview of Bridge3D. Our method employs features, semantic masks, and captions derived from foundation models to improve 3D representation learning. We use semantic masks to guide the masking and reconstruction phases in the masked autoencoder, which intensifies the network's attention on foreground objects. At the scene level, we use image captioning foundation models to bridge the scene-level 3D-text gap. Additionally, we facilitate the distillation of well- learned 2D and text representations the 3D model at the object level by leveraging foundation models to generate accurate object-level masks and semantic text information.
3. To the best of our knowledge, this is the first research to harness multiple foundation models for self-supprvised 3D scene understanding. This pioneering approach has been shown to outperform state-of-the-ar methods in various downstream tasks.
2 Related Work
3D Supervide Representation Learning. Recently, self-supervised pre-training on unla- belled point clouds [2: 24; 12; 66; 73; 35; 22] has shown promising transferable ability, providing good network initialization for downstream fine-tuning. Several methods have been proposed for pre-training point cloud features, including learning relative position [58], multiple pretext tasks [26], and contrastive learning [17; 33; 57; 65; 72; 3; 35; 23; 21]. Info3D [57] extends the InfoMax and contrastive learning principles 3D shapes. PointContrast [65] conducts point-level contrast two transformed views of the same point cloud. Zhang [72] contrasts instance-level representations obtained from the same scenario but processed by different model architectures. CrossPoint [3] introduces an auxiliary multi-modal contrastive objective that captures 3D-2D correspondence, lever- aging the complementary attributes of point clouds and images. Point-BERT [67] uses pre-trained tokenizers to indicate discrete point tokens, while Point-MAE [50] applies Masked Autoencoders (MAE) to directly reconstruct the 3D coordinates of masked tokens. Our proposed method uses Point-MAE as the baseline, but leverages foundation models to guide the masking and reconstruction stages. Additionally, we leverage image and text knowledge from foundation models to enhance 3D self-supervised learning.
Foundation Models. The field of AI research has experienced paradigm shift with the emergence of models trained on massive amounts of data at scale, commonly referred to foundation models [19; 56; 62; 25; 60]. These models have demonstrated remarkable performance in various language and visual-related tasks. The use of large-scale text pre-training attention- based models [15: 71] has led the increasing popularity of vision-language models (VLM) due their impressive performance in visual understanding tasks [54; 46; 40]. Recent advancements in contrastive learning have enabled CLIP [54] to perform multimodal learning with 400M data crawled from the web. CLIP has been extended for high-efficiency model training and cycle consistency through various methods

---

Window
Window
Bathnub
Sink
Toilat
(a) Semantic masks of MaskCLIP with all labels as prompts.
(b) Semantic masks of Ground-
(c) Semanntc masks Ground-
SAM with all labels prompts.
SAM with image caption prompts.
Figure 3: The comparison of zero-shot semantic results. (a) MaskCLIP failed to perform semantic segmentation accurately, yielding low accuracy overall (b) when all possible labels were used as prompts, the instance segmentation from foundation models results were prone to false positives (as seen in the Chair and Bathtub in this example) (c) by leveraging image captioning model to generates text prompts, the performance the instance segmentation from Ground-SAN further improved. This approach is beneficial for 3D scene understanding.
[40; 39; 38). BLIP [39] includes text-to-image generation as auxiliary task, which results in better performance by utilizing synthetic data as bonus. More recently, the success of the foundation models has been achieved in the pure computer vision area. Segment Anything (SAM) [37] has been proposed to act as a generic image segmentation model trained on the large visual corpus. To overcome the drawback of CLIP which overlooks the visual local information, DINOV2 [49] is proposed, which is trained with self-supervised learning and achieves results that match or surpasses the standard approach used in task task-specific fields. Grounding DINO [43] extends closed-set detector DINO to open-set set object detection by performing lision-language modality fusion at multiple phases.
Self-supervised 3D Understanding with Foundation Models. A number of studies have proposed strategies for knowledge transfer from pre-trained 2D foundation models to 3D representations at the object level [28; 30; 69; 68; 51]. For comprehensive scene understanding, recent efforts have improved 3D point representations by exploiting pixel-point alignments for distillation or contrastive learning [59; 9]. The 12P-MAE [70] approach takes advantage of 2D semantic saliency maps from CLIP [54] to guide masking and facilitate knowledge distillation from 2D to 3D at the instance level. Despite these advancements. most current 3D understanding methodologies employing foundation models focus predominantly on CLIP and distill knowledge through feature-level consistency or contrastive learning [9; 59]. The potential utilize the capabilities of other foundation models, such as BLIP [39] for image-to
captioning. SAM [37] for mask generation, Grounding DINO [43] for zero-shot detection, and DINOV2 [49] for high-performance features with detailed localized information, remains largely unexplored. Therefore, we propose to advance self-supervise 3D scene understanding by newly incorporating features, semantic masks, and captions obtained from various foundation models, achieving superior performance compared to state-of-the-a methodologies.
3 Methodology
The pipeline of Bridge3D is illustrated in Fig. 2. Our approach employs semantic information and features extracted from well-established foundation models to enhance 3D scene representation learning using self-supervised learning. The proposed method consists of three components: semantic-guided masked autoencoder, multi-modal scene-level knowledge distillation, and multi- modal object-level knowledge distillation.
3.1 Mask Generation by Foundation Models
Our proposed methodology leverages existing foundation models to produce instance segmentation masks. Initially, we use Tag2text [34], based on BLIP [39], to create image captions. We leverage ChatGPT filter captions objects neither ScanNet [14] nor SUN RGB-D [61] dataset generate text prompts for the 3D scene. Subsequently, employ SAM [37] to generate masks for the image.

---

(a) Masking results of SLIC super-pixels.
(b) Masking results by method which combines Girounding-DING and SAM.
Figure 4: The comparison of masks. (a) Masks generated from super-pixel and (b) Masks generated from our method. Our method can provide much more accurate object-level masks compared to super-pixel and thus benefits the multi-modality knowledge distillation.
Lastly, we use the text labels as prompts for Grounding Dino [43] to create the corresponding bounding box, which is then inputted into SAM to produce both zero-shot instance segmentation labels and segmentation masks ...... ON- To establish the dense visual token-point token correspondence Pi,Pi, we calibrate the point cloud with the respective images, where z; and p; signify the i paired image feature and point feature. This procedure is completed offline and saved locally, with the generated labels being used directly during the self-supervised training stage. As shown in Fig.3 and Fig.4, the instance segmentation masks generated from foundation models outperform previous methods terms of semantic results and object-level masks. Furthermore, Fig. 3 demonstrates that the performance of the foundation model is further improved when using caption methods as prompts and filtering out 3D-unrelated text.
3.2 Semantic Guided 3D Masked Autoencoder
To let the 3D model understand 3D-specific representations, we leverage Point-MAE [50] as the baseline for pre-training, which learns a meaningful representation with the pretext task of recovering the original inputs from visible ones. The Point-MAE [50] method utilizes standard Transformers as the backbone of its architecture, with an encoder-decoder structure that is asymmetric. The encoder takes visible tokens 7 as input and generates encoded tokens T while the decoder contains fewer Transformer blocks and responsible for generating the reconstructed masked 3D coordinates. Positional embeddings are incorporated into each Transformer block to provide location-base information. The encoded tokens T are padded with learnable mask tokens 7 and sent to the decoder. A set of positional embeddings is added to each Transformer block in the decoder to provide location information to all tokens. The output of the decoder H is then passed through a simple fully connected layer to reconstruct the masked 3D coordinates ppre. After that, it restores the coordinates of the points each masked point patch, and to evaluate the accuracy of the predicted coordinates, computes the reconstruction loss using l2 Chamfer Distance [18], which is formulated as:
Liae
Chamfer (ppre, pmaak)
Mmask
where Pmosk represents the ground truth of masked points.
Foreground-aware Masking and Patch Dropping. The Point-MAE [50] relies on random masking strategy, resulting in dispersed attention across the entire image and insufficient focus on foreground objects. This dispersion of attention can lead to model wasting computational resources on unimportant background elements, thereby leading to weaker learned representations. Moreover, the Point-MAE's approach to reconstructing all masked tokens, including those from the background, may further weaken representation learning the model overly concentrates on the
5

---

background due to the foreground-background imbalance. Although MAE [70] has suggested using 2D semantic saliency maps to guide the masking of point tokens for 3D instances in classification tasks, the problem of effectively guiding 3D scene masking still remains a key research challenge.
To address those problems, we propose semantic-guided masking strategy based on segmentation results obtained from foundation models. Specifically, for foreground objects obtained from the segmentation mentioned before, we mask higher percentage ry of foreground points compared to the whole masking ratio Tw. This generates a more challenging reconstruction task and forces the model to focus more on foreground objects. In addition, instead of inputting all patches (P:) as in Point-MAE, we randomly drop a percentage ra of background patches to obtain {Pa} Where N = ra) x M. The transformer decoder reconstructs masked point patches by using features from visible tokens and the positional information of both visible patches and mask patches. The patches that are dropped will not be reconstructed by the decoder. With enough background patches dropped, the decoder sees fewer points to perform the trivial up-sampling, which further improves the 3D representation learning and accelerates the pre-training by reducing the input data.
3.3 Scene-level Multi-modal Knowledge Distillation
Although some works have investigated scene-level multimodal learning using image and point clouds [36; 11; 41; 70], exploring scene-level multimodal learning with text and point clouds remains challenge due the lack of corresponding text descriptions for current 3D scene datasets. To address this problem, we propose to leverage image captioning methods to generate corresponding captions and filter out other 3D irrelevant objects to obtain 3D scene description texts f.. Then, we train the 3D network to align 3D point clouds with their corresponding scene-level images i, and texts t.. We formulate the proposed method below. Consider set of N point dloud-maage-text pairs { (1 F2D, Ftext), ), where F3D represents scene-level point cloud features from the encoder, FeD is the corresponding image features obtained from pre-trained foundation model based on is, and Frent is the text features from pre-trained foundation model based on l, Our model maps scene-level 3D features F3D to the hidden representation fmm for each modality with projection head Em. The mapping process can be formulated as:
Fm E, (F3D),
(2)
Due to the attributes to avoid representation collapsing, previous methods [9; 59] utilize InfoNCE loss [48] to conduct multi-modality knowledge distillation. However, in this work, we find that leveraging positive only L1 smooth loss generates better results. We think this is because those foundation models have learned discriminative features during the pre-training stage, and thus negative pairs are not necessary for the distillation stage. The scene-level distillation between 3D-image features; and 3D-text features are defined by:
Excene = (1122,F20) + L, (fteet plezt)
(3)
where I., represents the L. smooth loss.
3.4 Object-level Multi-modal Knowledge Distillation
While the value of object-wise feature representations downstream tasks like semantic segmentation and detection is well-proved [74; 29; 64; 6]. the generation of unsupervised masks presents a substan- tial challenge. Traditional techniques in computer vision, such as Hetrresswwabb-Huttenoooo
[20] and super-pixel [1], have been employed in earlier methods [59]. However, these techniques yield subpar masking results and cannot generate semantic labels, which hinders their ability to bridge the 3D-2D-text gap and leaves the potential of powerful language foundation models unrealized. The recent CLIP2Scene method [9] uses the MaskClip [76] model to generate dense semantic predic- tions, but it falls short of generating instance semantic results, which prevents object-level visual representations from being distilled into 3D models. The inferior quality outputs of MaskClip, hettensswwal--Httenloodd
and super-pixel impede 3D representation learning. In contrast, our proposed Bridge3D method leverages high-quality masks obtained from foundation models to guide object-level knowledge distillation, thereby enhancing 3D scene understanding.
In the object-level knowledge distillation phase of Bridge3I,, we propose a distinctive approach to multi-modality distillation through reconstructed tokens from the decoder, as opposed to previous

---

methods 19; 59] that directly distill the knowledge from other modalities to 3D post-encoder. The Point-MAE uses the decoder to reconstruct these masked tokens and learn specific features. In our method, we not only task the decoder with reconstructing masked point clouds but also reconstruct text and visual features that correlate with visible point tokens to distill the multi-modal knowledge. We do not reconstruct text and visual features of masked tokens that are dropped in the point reconstruction part, as mentioned in Sec. 3.2. To be specific, 1 represents the visual features obtained from the pre-trained visual model and belonging to the mask O, We use the mapping function to group the points into corresponding masks: ,I....,yk.. For each scene, we compute mask visual features and mask point features by average pooling:
(4)
iED;
1 Σ (H.)
RPP = 9
(5)
JCC.
Where H; represents visible tokens from the decoder,F is the projection head. For the text-to-poin cloud knowledge distillation, only foreground objects have the text information from the semantic labels, we choose corresponding foreground masks J1..... Is from g. Text features are obtained following:
Sts = Oreze frrst
(6)
J Σ (FtereeH))) JEJ.
Where i is the corresponding mask semantic label and Ótezt is the pre-trained text encoder, and Fremt is the projection head. We then transfer visual-text pairs to point-text pairs (f ftaeat and distill the knowledge from text to the point cloud in the object-level. The objective function is as follows:
K ELOOP
S fPP)+ EL
Cobject
Steet ftt stent)
(8)
K
The L the smooth L loss. Our final loss is the sum of previous loss terms.
Litnat Limae Locone + Lobject
Experiments
In this section, we first introduce the pre-training setting of Bridge3D. Then, we show the effectiveness of our method on several popular downstream tasks, including 3D object detection and 3D semantic segmentation. Finally, we conduct extensive ablation studies to show the effectiveness of each design. We put more details into the supplementary materials.
4.1 Self-supervised Pre-training
Network architectures. For the 3D backbone encoder, utilize the same architecture Point- MAE [50]. For the image branch, we follow DINOV2 ViT-B [49] to divide 518x518 images into regular patches with size of 37 x 37, before the ViT backbone. For the image branch, we directly utilize the CLIP ViT-B [54] to extract text features. For image captioning, we leverage Tag2text [34].
Pre-training. During this stage, we perform training the model for 120 epochs by employing the ScanNet dataset [14] consisting of point clouds and their corresponding images. For the text prompts, we only utilize all class names of ScanNet and SUN RGB-D as the prompts and filter other classes generated from image captioning. We use Adam W [45] optimizer with a base learning rate of 5e-4 and weight decay 5e-2, along with batch size of 64. The whole masking ratio ra is set to 70% and the drop ratio ra is set to 40%. The cosine learning rate scheduler is applied, with drop path rate and warm-up epochs set to and 10, respectively. The encoder depth is set to and we utilize the same decoder as Point-MAE [50], with the decoder depth set to 2.
7

---

SUN RGB-D
ScanNetV2
Methods
Pre-trained
AP25
APso
AP
APso
VoteNet [52]
None
57.7
32.9
58.6
33.5
PointContrast [65]
57.5
34.5
59.2
38.0
Hou al. [32]
36.4
39.3
4DContrast [10]
38.2
40.0
DepthContrast [72]
61.6
35.5
64.0
42.9
DPCo [41]
60.2
35.5
64.2
41.5
3DETR [47]
None None
58.0
30.3
62.1
37.9
+Bridge3IDffrom scratch)
57.6
31.9
61.1
38.6
PPint-BERT167
61.0
38.3
+Point-MAE [50]
63.4
40.6
+MaskPoint [42]
63.4
40.6
+ACT [16]
63.5
41.0
+PiMAE [8]
59.9
33.7
63.0
40.2
+Bridge3D
61.8(+3.8)
35.9(+5.6)
65.3(+3.2)
44.2(+6.3)
GroupFree3D [44]
None
63.0
45.2
67.3
48.9
+(rro scratch)
None
62.2
45.0
66.1
48.3
+Point-MAE [50]
63.9
46.1
67.4
49.8
+PiMAE [8]
65.0
46.8
67.9
50.5
+Bridge3D
67.9(+4.9)
48.5(+3.3)
69.1(+1.8)
51.9(+3.0)
Table 3D object detection results on ScanNet and SUN RGB-D dataset. We adopt the average precision with 3D IoU thresholds of 0.25 (AP25) and 0.5 (APto) for the evaluation metries.
S3DIS
ScanNetV2
Methods
Pre-trained
mIoU
m.cc
mIoU
m.Acc
SR-UNet [65]
None
68.2
75.5
72.1
80.7
PointContrast [65]
70.9
77.0
74.1
81.6
DepthContrast [72]
70.6
73.1
Hou et al. [32]
/
72.2
73.8
Standard Transformer [67]
None
60.0
68.6
PointBert [67]
/
60.8
69.9
PVTT [53]
None V
64.4
69.9
PViT+Pix4Poin [53]
69.6
75.2
Ours(from scratch)
None
61.1
67.2
67.3
73.1
+Point-MAE [50]
64.8
70.2
+Bridge3D
70.2 (+9.1)
76.1(+8.9)
73.9(+6.6)
80.2(+7.11
Table 2: 3D semantic segmentation results on S3DIS dataset. We adopt the mean accuracy (mAcc) and mean IoU (mloU) for the evaluation metrics.
4.2 Results on Downstream Tasks
For fine-tuning downstream tasks, we discard decoders in pre-training and append task-specific decoders onto the encoder for different tasks.
Object Detection. To demonstrate the generality of the proposed method, we also pre-train it on the indoor Sca dataset [14] and subsequently fine-tune our method on the object detection task in ScanNetV2 dataset and SUN RGBD [75]. We report our performance on indoor 3D detection based on SOTA methods 3DETR [47] and GroupFree3D [44]. The Table indicates that Our method achieves 66.3 AP25 (+4.2) and 45.5 AP (+7.6) compared to the baseline 3DETR on the ScanNetV2 dataset and also brings significant improvements to both models, surpassing previous baselines consistently in all other datasets and criteria. These experiments' results showcase method's

---

Point-
Semantic Guided
Scene-level Distillation
Object-level Distillation
ScanNetV2
S3DIS
MAE
Reconstruction
AP
APto
mloU
m.Acc
62.3
39.9
64.8
70.2
63.2
41.1
66.2
71.1
64.4
43.0
68.4
73.7
65.3
44.2
70.2
76.1
Table 3: The effectiveness of each component. Ablation study on the effectiveness of each component on 3D object detection and semantic segmentation tasks.
Scan
S3DIS
Text
Image
AP25
AP30
mIoU
mAcc
62.1
37.9
61.1
67.2
64.2
42.5
67.8
74.1
64.7
43.3
68.3
74.5
65.3
44.2
70.2
76.1
Table 4: The effectiveness of each modality. Ablation study on the effectiveness of each modality on 3D object detection and semantic segmentation tasks.
effectiveness in learning superior 3D representations for object detection, highlighting its potential to benefit a wide range of 3D applications.
Semantic Segmentation. In Tab. 2, present the semantic segmentation results the S3DIS dataset. Despite their efforts, prior 3D self-supervised methods such as PointContrast(65) and [72] only achieved marginal improvements post-pre-traninng (+2.7 and +2.4 in mIoU). Conversely, Pix4Point [53], utilizing a pre-trained 2D model, demonstrated significant progress compared to training from scratch. Most notably, our proposed method incorporates multiple foundation models during pre-training, elevating the metrics by 10.0 and 10.3 respectively, markedly surpassing other state-of-the-art 3D self-supervised methods. These results substantiate the effectiveness of utilizing multiple foundation models for enhancing 3D representation learning for semantic segmentation.
4.3 Ablation Studies
The effectiveness of Each Component. As shown in Table 3, the results indicate that each component, including the foreground-award masking strategy, multi-modal scene-level knowledge distillation, and multi-modal object-level knowledge distillation, contributes better results. More- over, when we combined all components, our proposed method achieved the best performance. The foreground-aware masking strategy proved to be important enhanced the learning of foreground object representations in the 3D masked autoencoder. The multi-modal scene-level knowledge distil- lation, which leverages image captioning model to generate text descriptions from paired images of point clouds, helped bridge the gap between 3D and text at the scene level. The multi-modal object-level knowledge distillation, which uses foundation models to generate accurate object-level masks and semantic text information, bridged the gap between 3D, 2D, and text object-level fea- tures. Overall, our ablation study demonstrates the effectiveness of each component our proposed framework and highlights the importance of leveraging foundation models to improve 3D scene representation learning.
The effectiveness of Each Modality. Table 4 provides clear insight into the significant contri- bution of each modality the overall performance our method. By integrating all modalities, our method realizes its full potential, showcasing the best results. However, it's worth noting the inherent resilience our method exhibits to modality variations. This adaptability implies that even when the modality mix is altered or reduced, the system remains relatively unaffected in terms of its output quality. This inherent resilience not only underscores the robust architecture of our model but also offers users the freedom to customize the framework based on their specific requirements. The adaptable nature modality inclusion thus ensures that our method remains both versatile and efficient, enabling users to balance computational overhead with optimal performance.

---

ScanNetV2
S3DIS
Semantic Mask Method
2P
APoo
mloU
mAAc
Point-MAE [50]
62.3
39.9
64.8
70.2
MaskCLIP [76] Superpixels [1]
64.1
43,0
67.1
73.4
Bridge3D
65.3
44.2
70.2
76.1
Table 5: The effectiveness of mask. Ablation study on unsupervised mask methods on 3D object detection and semantic segmentation tasks.
SUN RGB-D
ScanNetV2
Methods
Pre-trained
AP25
APso
AP25
APso
CAGroup3D [63]
None
66.8
50.2
75.1
61.3
+Bridge3D (scene & object level distillation)
68.7
52.1
76.3
62.2
Table 6: The performance on SOTA method. 3D object detection results on ScanNet and SUN RGB-D dataset based on CAGroup3D.
The Effectiveness of Masks from Foundation Models. In Table 5, conduct comparative study of mask generation strategies employed by our method and those utilized by prior works. For instance, CLIP2Scene(9] employs MaskCLIP [76] to produce semantic masks. However, this approach fails to provide instance segmentation masks and is solely capable of guiding text-to-3D knowledge distillation. Alternatively, SLidR [59] leverages superpixels [1] for generating object-level masks. Despite this, the superpixels lack semantic information, limiting their use to guiding 2D-to-3D knowledge distillation. In contrast, our method generates instance semantic information, enhancing the functionality and accuracy of the masks. For a comparative analysis, we combine superpixels and MaskCLIP to direct both 2D-to-3D and text-to-3D distillation, essentially mimicking the fusion of CLIP2seene and SLidR. The experimental results reveal that masks generated through our method, which leverages foundation models, yield substantial improvements compared the combined use of MaskCLIP and superpixels.
Apply Bridge3D in SOTA Method. Our method's s adaptability extends to its successful application to the recent state-of-the-arr (SOTA) work, CAGroup3I) [63]. While the network structure made a direct application of the plain transformer challenging, we devised unique adaptation, pre-training the CAGroup3D'ss backbone with our scene and object-level distillation, excluding the reconstruction part. Table illustrates how our pre-training approach can enhance and benefit current SOTA 3D detection methodologies, showcasing the potential reach of our technique. Importantly, it should be noted that our framework is optimized for plain mansformer-based backbones, and thus the application of Bridge3D to alternative backbones may lead to a reduction performance.
5 Conclusion
In this work, we introduce a pioneering method Bridge3I that capitalizes on foundation models to overcome the hurdles in self-supervised 3D learning. This innovative approach not only enables more focused learning on foreground object representation but crucially bridges the domain gap between 3D and text at the scene level. Furthermore, it enriches the quality of 3D scene representation learning by generating highly accurate object-level masks and semantic textual information, effectively bridging the gap between 3D, 2D, and text object-level features. Our comprehensive experimental results corroborate the superior effectiveness of our approach in amplifying 3D scene understanding. However, the current work primarily focuses on indoor scene understanding, which constitutes a limitation. Looking ahead, we plan to broaden the applicability of Bridge3D to encompass more diverse set tasks, including outdoor scene understanding and open- vocabuooyyy 3D tasks. Our work is expected to have negative socictal implications.
10

---

References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, (4(11))2274-2 222222222222. 2012.
[2] Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning rep- resentations and generative models for 3d point clouds. arXiv preprint 17000:1707.02332..
2017.
[3] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-super cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9902-9912 992,, 2022.
[4] Jean- -an--Btttstte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237336 2022.
[5] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of scage-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534-1543, 2016.
[6] Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level
region contrast for object detection pre-training. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 16061-16070, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems. 13:1877-,901, 2020.
[8] Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang. Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. arXiv preprint arXiv:2303.0 081 29,, 2023.
[9] Rumnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301. 04926, 2023.
[10] Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Parl XXXII, pages 543-560. Springer, 2022.
[11] Zhimin Chen, Longlong Jing, Yang Liang. YingLi Tian, and Bing Li. Multimodal semi- supervised learning for 3d objects. arXiv preprint arXiv:2 10. 1601, 2021. [12] Zhimin Chen, Longlong Jing. Liang Yang. Yingwei Li, and Bing Li. Class-level confidence based 3d semi-supervised learning. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 633-642, 2023.
[13] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXXv:2204.023/1 2022.
[14] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
Nießner. Scannet: Richly-annotale
[15] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1xXiv:1800.045) 04805, 2018.
[16] Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:22/2.08322 2022.
[17] Bi'an an Du, Xiang Gao, Wei Hu, and Xin Li. Self-contrastive learning with hard negative sampling for self-supervised point cloud learning. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3133-3142, 2021.
[18] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[19] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint Xrxiv:2211.0077 2022.

---

[20] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based based image segmentation.
International journal of computer vision, 59:167-181, 2004.
[21] Ziyue Feng, Longlong Jing. Peng Yin, Yingli Tian, and Bing Li. Advancing self-supervised monocular depth learning with sparse lidar. In Conference on Robot Learning. pages 685-694.
2022.
[22] Ziyue Feng, Liang Yang, Pengsheng Guo, and Bing Li. Cvrecon: Rethinking 3d geometric feature learning for neural reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17750-17760, 2023.
[23] Ziyue Feng, Liang Yang, Longlong Jing. Haiyan Wang. YingLi Tian, and Bing Li. Disentangling object motion and occlusion for unsupervised multi-frame monocular depth. In European Conference on Computer Vision, pages 228-244. Springer, 2022.
[24] Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference on Computer Vision (ECCV),
pages 103-118, 2018
[25] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all, 2023.
[26] Kaveh Hassani and Mike Haley. Unsupervised multi-task feature learning on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8160-8171, 2019.
[27] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7729-9738, 2020.
[28] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303.1131 2023. [29] Olivier Hénaff, Skanda Koppula, Jean- aan-ttppistt Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10086-10096, 2021.
[30] Georg Hess, Adam Tonderski, Christoffer Petersson, Lennart Svensson, and Kalle Åström. Lidarclip or: How i learned to talk to point clouds. arXiv preprint arXiv:22 22. 2.06858, 2022.
[31] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Flena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:22033 /5556, 2022.
[32] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15587-15597. 2021. [33] Siyuan Huang, Yichen Xie, Song-Chun representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535-6545, 2021.
Chun Zhu, and Yixin Zhu. Spatio-temporal self- self-supervised
[34] Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprini arXiv:2303.088 0003,05657,2200 2023.
[35] Longlong Jing, Yucheng Chen, Ling Zhang, Mingyi He, and Yingli Tian. Self-supervised modal and view invariant feature learning. arXiv preprini arXiv:2005. /4169, 2020.
[36] Longlong Jing, Zhimin Chen, Bing Li, and Yingli Tian. Self-supervised modality-invaiia and modality-specific feature learning for 3d objects, 2022.
[37] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,
Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprins arXiv:2304. 02643, 2023.
[38] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[39] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[40] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 0496694-7703 2021. [41] Lanxiao Li and Michael Heizmann. A closer look at invariances in self-supervised pre-training for 3d vision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXX, pages 656-673. Springer, 2022.
12

---

[42] Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self- supervise learning on point clouds. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part 11, pages 657-675. Springer, 2022.
[43] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303..m 05499, 2023.
[44] Ze Liu, Zheng Zhang, Yue Cao. Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 2949, 2021.
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:/711.0511 /01, 2017.
[46] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for Dision-and-language tasks. Advances in neural information processing systems, 32, 2019.
[47] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906-2917, 2021.
[48] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 8377003748 2018
[49] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07II
2023. [50] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self supervipe learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I1, pages 604-621, Springer, 2022.
[51] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211. 156544, 2022.
[52] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9277-9286, 2019.
[53] Guocheng Qian, Xingdi Zhang, Abdullah Hamdi, and Bernard Ghanem. Pix4point: Image pretrained transformers for 3d point cloud understanding. arXiv preprint arXiv:2208. 12259, 2022.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8766 PMLR, 2021.
[55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 (5:5555555555 2020. [56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical lext-conditional image generation with clip latents. arXiv preprint 2rXiv22044.6/255 2022. [57] Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In European Conference on Computer Vision, pages 626-642. Springer, 2020.
[58] Jonathan Sauder and Bjarne Sievers. Self deep learning on point clouds by recon- structing space. Advances in Neural Information Processing Systems, 32, 2019.
[59] Corentin Sautier, Gilles Puy. Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidaal supervied distillation for autonomous driving data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9891-9901, 2022.
[60] Mannat Singh. Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, et al. The effec- tiveness of mae preteetrainn for billion-scale pretraining. arXiv preprint arXiv:2303. /3496. 2023.
[61] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
13

---

[62] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[63] Haiyang Wang. Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li. Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. Advances in Neural Information Processing Systems, 1--29755-2988 2022.
[64] Fangyun Wei, Yue Gao, Zhirong Wu. Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. Advances in Neural Information Processing Systems, 34:22682-2222222 22694,, 2021.
[65] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European conference computer vision, pages 574-591. Springer, 2020.
[66] Yaoging Yang, Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 206-215, 2018.
[67]
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-train training 3d point cloud transformers with masked point modeling. In Proceedings of the
Computer Vision and Pattern Recognition, pages 19313-19322 2022. [68] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open- 3d scene representations from 2d dense clip, arXiv preprint arXiv:2303 230.00448.44 2023.
IEEE/CVF Conference
[69] Renrui Zhang. Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 8552 8562, 2022. [70] Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d represen- tations from 2d are-trained models via image-to-pal to-point masked autoencoders. arXiv preprint arXiv:22/2.0613 2022.
[71] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained
transformer language models. arXiv preprint arXiv:2205, 01068, 2022.
[72] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of
3d features on any point-cloud, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252-10263, 2021.
[73] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1009-1018, 2019.
[74] Yiwu Zhong, Jianwei Yang, Pengehuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based lunguage-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16805. 2022.
[75] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
[76] Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXi::2112001071, 2021.
14


Reviews:
[{"review_id": "bg5E1jaWtY", "reviewer": "Reviewer_V9nH", "summary": "This paper proposes a methodology named Bridge3D to pre-train 3D models using features, semantic masks, and captions sourced from foundation models. Experiments have been conducted on different 3D understanding frameworks.", "questions": "The reviewer is curious whether the authors have tried to simplify this framework.\nAnother question is that, for the 3D object detection results, table 1, it seems only the basic versions of the models have been tried. For example, the GroupFree3D, the best performance on ScanNetV2 reported in this paper is 69.7 and 52.4, however, in the GroupFree3D original paper, the best performance they reported is 69.1 and 52.8, which is basically on par with the reported results in this paper, the 3DETR results have the same issue. Have the authors tried to apply your method to their most powerful models? What is the highest performance you may get in that case, will that reach new SOTA results? So far the improvement compared to the vanilla baseline looks fine but in order to demonstrate the proposed method to be more useful, the absolute performance or the highest performance by applying the proposed method needs to be explored.", "limitations": "Refer to the weakness part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The overall architecture seems to be reasonable; using the modern foundation models and multimodal supervision is promising to help 3D understanding, and some of the results show a good improvement gap compared to training from scratch.", "weaknesses": "This framework seems to be overall complicated, which might involve quite some tuning effort and impact its scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "With gratitude for the reviewer's valuable insights, we will make appropriate revisions to our paper. In the following section, we address each of the reviewer's questions and comments in detail.\n\n**W1: The complexity of the framework.**\n\n**Concerns on complexity.**\nThe complexity observed in our approach is primarily due to our full utilization of modern foundation models across both image and text modalities. This integration is vital to capture the intricate relationships inherent in multi-modal modeling and fully utilize foundation models for 3D representation learning. However, the complexity is not an immovable feature of our model. Our framework consists of three distinct components and two teacher modalities, all designed independently. The ablation study in Table 3 of the main paper reveals the positive contribution of each component to the final results, justifying the flexibility. While this robust design may initially seem daunting, it is strategically crafted to prevent complexity from becoming a barrier. For specific use cases, users have the flexibility to select and combine different components and modalities, enabling them to harness the full capabilities of our method or decrease complexity according to their requirements. As shown in Figure 2 in the main paper, it is convenient to decrease modalities or components by just removing the corresponding distillation loss. Our method is a powerful and flexible framework capable of achieving state-of-the-art results without undue complexity or difficulty in implementation.\n\n**Concerns on tuning efforts.** Our approach is not as tuning-intensive as it may appear. Many elements of mask and caption generation rely on pre-existing foundation models that are frozen and require no additional tuning. In terms of training, our method only introduces one additional parameter related to the foreground masking ratio. This extra parameter does not significantly impact scalability, particularly when compared to the baseline method, Point-MAE.\n\n**W2: Apply Bridge3D to more powerful models.**\n\n**1, Compare with the best performance of GroupFree3D.** We explored our method's efficiency by adjusting the plain transformer to ViT-B configurations, allowing a direct comparison with the PointNet++x2 settings in GroupFree3D. Our novel approach led us to surpass GroupFree3D's performance with PointNet++w2× settings, as shown in Table 1. Remarkably, the results on the SUN RGB-D dataset have reached **state-of-the-art** performance. Though we are optimistic that using ViT-L configurations could further elevate our performance in ScanNet, our resources' limitations have hindered such experimentation for now.\n\n**Table 1: 3D object detection results on ScanNet and SUN RGB-D dataset based on GroupFree3D.**\n\n| Methods              | Encoder        | Pre-trained | SUN RGB-D $AP_{25}$ | SUN RGB-D $AP_{50}$ | ScanNetV2 $AP_{25}$ | ScanNetV2 $AP_{50}$ |\n|----------------------|----------------|-------------|---------------------|---------------------|---------------------|---------------------|\n| GroupFree3D          | PointNet++w2×  | *None*      | -                   | -                   | 69.1                | 52.8                |\n| +plain transformer   | ViT-B          | *None*      | 65.2                | 47.1                | 68.6                | 51.9                |\n| +Bridge3D            | ViT-B          | ✓           | **69.6**            | **52.6**            | **71.2**            | **54.4**            |\n\n**2, Apply Bridge3D in SOTA methods.**  Our method's adaptability extends to its successful application to the recent state-of-the-art (SOTA) work, CAGroup3D [1]. While the network structure made a direct application of the plain transformer challenging, we devised a unique adaptation, pre-training the CAGroup3D's backbone with our scene and object-level distillation, excluding the reconstruction part. Table 2 illustrates our pre-training approach can enhance and benefit current SOTA 3D detection methodologies, showcasing the potential reach of our technique. Importantly, it should be noted that our framework is optimized for plain transformer-based backbones, and thus the application of Bridge3D to alternative backbones may lead to a reduction in performance.\n\n**Table 2: 3D object detection results on ScanNet and SUN RGB-D dataset based on CAGroup3D.**\n\n| Methods                                               | Pre-trained | SUN RGB-D $AP_{25}$ | SUN RGB-D $AP_{50}$ | ScanNetV2 $AP_{25}$ | ScanNetV2 $AP_{50}$ |\n|-------------------------------------------------------|-------------|-----------|-----------|-----------|-----------|\n| CAGroup3D [1]                                         | *None*      | 66.8      | 50.2      | 75.1      | 61.3      |\n| +Bridge3D (scene & object level distillation)         | ✓           | **68.7**  | **52.1**  | **76.8**  | **62.6**  |\n\n[1] Wang, Haiyang, et al. \"Cagroup3d: Class-aware grouping for 3d object detection on point clouds.\" Advances in Neural Information Processing Systems 35 (2022): 29975-29988.\n\n\n\n"}, {"review_id": "bk3IxVYbkx", "reviewer": "Reviewer_XLcq", "summary": "This paper proposes a combination of several techniques to pre-train 3D models by leveraging 2D foundation models. Specifically, the paper proposes: a) an MAE pre-training guided by foreground masks generated by foundation models to mask the foreground masks more compared to the background, to encourage the models to focus more on the foreground. b) Scene-level feature distillation between 3D-image and 3D-text with the help of image captioning foundation model c) Object level feature distillation with the help of mask generated via foundation models. The results show better performance compared to prior methods of self-supervising the 3D models and the ablations confirm the benefit of each proposed technique.", "questions": "N/A, I do any have any specific questions.", "limitations": "No, I think it would be useful to specify limitations in the main paper (for eg: complexity of the approach)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well-written and the results are impressive compared to the baselines. The ablations are clear and well-motivated. Investigating how far we can get by leveraging 2D foundation model is an important research direction, and this paper does a good job in exploring several ways to go about it. ", "weaknesses": "I do not have any substantial complaint with this paper, however, I think the main limitation is in the complexity of the proposed technique: the approach has several moving components relying on several off-the-shelf models and it seems like a combination of a bunch of techniques to utilize the 2D foundation models; requiring several offline computation of intermediate results. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "With gratitude for the reviewer's valuable insights, we will make appropriate revisions to our paper. In the following section, we address each of the reviewer's questions and comments in detail.\n\n**W1: The complexity.**\n\n**1, Complexity of the framework.** The complexity observed in our approach is primarily due to our full utilization of modern foundation models across both image and text modalities. This integration is vital to capture the intricate relationships inherent in multi-modal modeling and fully utilize foundation models for 3D representation learning. However, the complexity is not an immovable feature of our model. Our framework consists of three distinct components and two teacher modalities, all designed independently. The ablation study in Table 3 of the main paper reveals the positive contribution of each component to the final results, justifying the flexibility. While this robust design may initially seem daunting, it is strategically crafted to prevent complexity from becoming a barrier. For specific use cases, users have the flexibility to select and combine different components and modalities, enabling them to harness the full capabilities of our method or decrease complexity according to their requirements. As shown in Figure 2 in the main paper, it is convenient to decrease modalities or components by just removing the corresponding distillation loss. Our method is a powerful and flexible framework capable of achieving state-of-the-art results without undue complexity or difficulty in implementation.\n\n**2, Complexity of the computation.** It's worth noting that although our method incorporates several off-the-shelf foundation models to obtain masks, captions, and features, this complexity is somewhat mitigated by the nature of our approach. The computation involving these models is performed offline and only once. During both the training and inference stages, these foundation models are not used, meaning they do not substantially increase the complexity of computation. Such preprocessing is common in self-supervised learning methods like SLidR [53] and CLIP2Scene [9] mentioned in the main paper. Thus, while the initial setup may seem complex, it is in line with established methodologies and does not unduly burden the ongoing computational process. We believe that this carefully designed architecture enables us to harness the power of these foundation models without unnecessary complications in deployment.\n\n**W2: Limitations not discussed.** \n\nThank you for your thoughtful suggestion. We acknowledge that a comprehensive discussion of the limitations of our approach, including its complexity, is paramount to a well-rounded understanding. In the following section, we will delve into these limitations and outline potential avenues for future work.\n\nWhile our work has achieved promising results in 3D detection and segmentation tasks, it encounters specific limitations that merit further exploration. A significant constraint is the method's reliance on existing foundation models for mask and caption generation, leading to several offline computations of intermediate results and time-consuming data preprocessing. This dependency also means that changing or updating these underlying models might affect overall performance. In the future, we aim to decrease the computation complexity by integrating high-speed models, such as mobile SAM [1], and create a more balanced and efficient system that can maintain high performance without being heavily tied to the pre-existing models currently in use.\n\n[1] Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., ... & Wang, J. (2023). Fast Segment Anything. arXiv preprint arXiv:2306.12156.\n"}, {"review_id": "qHXebzy1H9", "reviewer": "Reviewer_E1Dc", "summary": "a novel masking and patch-dropping strategy based on foundation models to refine the focus of the network on foreground representation learning for 3D masked autoencoders. \n\na novel scene-level and object-level multi-modality knowledge distillation method that pre-trains a 3D network via features, semantic masks, and captions obtained from foundation models", "questions": "Misc:\nThe *Mask*s mentioned in 3D representation and in 2D segmentation are two different things and may seem confusing. It's better if you could replace one of them by other words (like occupancy for 3D)", "limitations": "see weaknesses", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Personally I really like the self-supervised framework. It's natural to use MAE to reconstruct and build 3D representations, and obtaining 3D features from 2D images and text is also reasonable. The segmentation results seem good.\n", "weaknesses": "* Lack of qualitative examples. The biggest problem to me is that no qualitative examples are presented in the paper (and visualization is really important for 3D reconstruction and segmentation!)\n* The experiments focus on segmentation. However, there's more to do with foundation models - like showing zero-shot results, captioning (generation or retrieval) etc. \n* The paper doesn't mention several recent papers on 3D representation / feature / concept learning from 2D features (like CLIP). Just to name a few:\n\nConceptFusion: Open-set Multimodal 3D Mapping\n\nSemantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models \n\n3D Concept Learning and Reasoning from Multi-View Images\n\n\nThis paper is also really really similar:\n\nULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding\n\nAlso, I think the self-supervision and feature learning framework shares merits with this paper:\n3D Concept Grounding on Neural Fields.\n\nSome of these methods could serve as baselines for segmentation. If not, you *should at least cite a few papers in the domain*.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "With gratitude for the reviewer's valuable insights, we will make appropriate revisions to our paper. In the following section, we address each of the reviewer's questions and comments in detail.\n\n \n\n**W1: Lack of visualization results**\n\nThank you for the valuable feedback regarding the absence of visualization for segmentation and reconstruction in our paper. We put visualization results of reconstruction, segmentation, and detection in the attached PDF file in the global response.\n\n\n**W2: More to do with foundation models**\n\nThank you for your insightful feedback. Currently, our focus is on leveraging foundation models to advance 3D detection and segmentation. While we recognize the potential in broadening our approach to encompass areas such as 3D zero-shot results and captioning, the constraints of the rebuttal period prevent us from diving into these tasks immediately. In the future, we will explore those avenues. Your perspective is invaluable and provides clear direction for expanding the horizons of our research. We're grateful for your thoughtful suggestions.\n\n\n**W3: Cite more papers**\n\nThank you for bringing to our attention the absence of references to recent papers on 3D representation, feature, and concept learning from 2D features. \n\n**1, Citation.** We acknowledge that these works form important contexts and could provide additional insights into our work. Currently, our focus is primarily on leveraging foundation models for 3D detection and segmentation, and it appears we inadvertently omitted some relevant recent developments. We are grateful for your feedback, and we will add and discuss them in relative works.\n\n\n**2, Comparison.** It's worth noting that we have included a comparison with a recent CVPR 2023 paper, CLIP2Scene [9], in our ablation study. This paper proposes the use of MaskCLIP [69] to generate 2D masks and distill corresponding object-level text features into 3D, which is similar to the paper Semantic Abstraction [2]. By applying it to our framework and comparing it, we found that our method outperforms it in both 3D detection and segmentation tasks.\n\n**3, Difference to the ULIP [1].** The ULIP method is specifically tailored for pre-training on instance-level datasets such as ModelNet40 and ScanObjectNN, and it is hard to be directly applied to 3D scene understanding tasks. Furthermore, our method overcomes the limitations of ULIP that need text labels to generate text features. We leverage powerful image captioning models to generate scene text descriptions and utilize CHATGPT to refine these texts without requiring annotations. \n\n\n**W4: Replace 2D/3D masks by other words**\n\nThank you for pointing out the potential confusion regarding the use of the term \"Masks\" in both 3D representation and 2D segmentation within our manuscript. Your observation highlights an important subtlety that we may have overlooked. We will differentiate these concepts by using distinct terms, 3D occupancy in the final version. \n\n[1] Xue, Le, et al. \"ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[2] Ha, Huy, and Shuran Song. \"Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models.\" 6th Annual Conference on Robot Learning. 2022."}, {"review_id": "UbN50TolhE", "reviewer": "Reviewer_hzsQ", "summary": "This paper aims to improve the pre-training of 3D point cloud models by leveraging various foundational models, including vision models such as SAM and Grounding Dino, language models like ChatGPT, and vision-language models. To achieve this objective, the paper proposes several techniques:\n1. Generating instance segmentation masks for the corresponding 2D images of 3D point clouds. This is accomplished by first generating image captions using caption models and language models like Tag2text and ChatGPT. Then, vision models such as SAM and Grounding Dino are employed to generate masks based on these captions.\n2. Masked point cloud reconstruction with the masks obtained in the previous step.\n3. Scene-level distillation using the features of an image encoder (2D-3D Disti) and a language encoder (Text-3D Distill).\n4. Object-level distillation using the features of an image encoder (2D-3D Disti) and a language encoder (Text-3D Distill).\nThe proposed pre-training method is evaluated on several downstream tasks, including ScanNetV2 and SUN-RGBD object detection, as well as S3DIS semantic segmentation.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of utilizing foundational models for 3D point cloud modeling is both interesting and promising.\n2. The paper is well-written and easy to follow.\n3. The experimental results consistently demonstrate improvements in several downstream tasks when compared to the baseline.", "weaknesses": "1. My main concern is that the experimental results are not sufficiently convincing. Although the results demonstrate consistent improvements compared to the baselines, the performance of the baselines used in this paper appears to be relatively low (e.g., the results of 3DETR and GroupFree3D). Additionally, the final system's performance lags behind the current state-of-the-art methods.\n2. The overall pipeline is complex, making it difficult to evaluate the effectiveness of each component, such as the model selection for each component.\n3. typos: (1) On line 159, the reference 'MAE [63]' is incorrect. (2) On line 233, it should be 'text branch' instead of 'image branch' when mentioning the utilization of CLIP ViT-B [48] to extract text features. (3) The teaser of Figure 2 is somewhat confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "With gratitude for the reviewer's valuable insights, we will make appropriate revisions to our paper. In the following section, we address each of the reviewer's questions and comments in detail.\n\n \n**W1: Experimental results are not sufficiently convincing**\n\n**1, Compare with the best performance of GroupFree3D.** We explored our method's efficiency by adjusting the plain transformer to ViT-B configurations, allowing a direct comparison with the PointNet++x2 settings in GroupFree3D. This novel approach led us to surpass GroupFree3D's performance with PointNet++w2× settings, as evident in Table 1. Remarkably, the results on the SUN RGB-D dataset have reached **state-of-the-art** performance. Though we are optimistic that using ViT-L configurations could further elevate our performance in ScanNet, our resources' limitations have hindered such experimentation for now.\n\n**Table 1: 3D object detection results on ScanNet and SUN RGB-D dataset based on GroupFree3D.**\n\n| Methods              | Encoder        | Pre-trained | SUN RGB-D $AP_{25}$ | SUN RGB-D $AP_{50}$ | ScanNetV2 $AP_{25}$ | ScanNetV2 $AP_{50}$ |\n|----------------------|----------------|-------------|---------------------|---------------------|---------------------|---------------------|\n| GroupFree3D          | PointNet++w2×  | *None*      | -                   | -                   | 69.1                | 52.8                |\n| +plain transformer   | ViT-B          | *None*      | 65.2                | 47.1                | 68.6                | 51.9                |\n| +Bridge3D            | ViT-B          | ✓           | **69.6**            | **52.6**            | **71.2**            | **54.4**            |\n\n**2, Apply Bridge3D in SOTA methods.**  Our method's adaptability extends to its successful application to the recent state-of-the-art (SOTA) work, CAGroup3D [1]. While the network structure made a direct application of the plain transformer challenging, we devised a unique adaptation, pre-training the CAGroup3D's backbone with our scene and object-level distillation, excluding the reconstruction part. Table 2 illustrates our pre-training approach can enhance and benefit current SOTA 3D detection methodologies, showcasing the potential reach of our technique. Importantly, it should be noted that our framework is optimized for plain transformer-based backbones, and thus the application of Bridge3D to alternative backbones will lead to a reduction in performance.\n\n**Table 2: 3D object detection results on ScanNet and SUN RGB-D dataset based on CAGroup3D.**\n\n| Methods                                               | Pre-trained |SUN RGB-D $AP_{25}$ | SUN RGB-D $AP_{50}$ | ScanNetV2 $AP_{25}$ | ScanNetV2 $AP_{50}$ |\n|-------------------------------------------------------|-------------|-----------|-----------|-----------|-----------|\n| CAGroup3D [1]                                         | *None*      | 66.8      | 50.2      | 75.1      | 61.3      |\n| +Bridge3D (scene & object level distillation)         | ✓           | **68.7**  | **52.1**  | **76.8**  | **62.6**  |\n\n**W2: The complexity of the framework**. \n\nThe complexity observed in our approach is primarily due to our full utilization of modern foundation models across both image and text modalities. This integration is vital to capture the intricate relationships inherent in multi-modal modeling and fully utilize foundation models for 3D representation learning. However, the complexity is not an immovable feature of our model. Our framework consists of three distinct components and two teacher modalities, all designed independently. The ablation study in Table 3 of the main paper reveals the positive contribution of each component to the final results, justifying the flexibility. While this robust design may initially seem daunting, it is strategically crafted to prevent complexity from becoming a barrier. For specific use cases, users have the flexibility to select and combine different components and modalities, enabling them to harness the full capabilities of our method or decrease complexity according to their requirements. As shown in Figure 2 in the main paper, it is convenient to decrease modalities or components by just removing the corresponding distillation loss. Our method is a powerful and flexible framework capable of achieving state-of-the-art results without undue complexity or difficulty in implementation.\n\n**W3: Typos**\n\nThank you for meticulously identifying the specific typos and inconsistencies in our manuscript. Your attention to detail greatly aids in enhancing the accuracy and clarity of our work. Regarding the MAE [63] reference on line 159, we will correct it as I2P-MAE  in the revised version. On line 233, your observation is indeed accurate. It should be 'text branch' instead of 'image branch' when referring to the utilization of CLIP ViT-B [48] for text feature extraction. We will amend this error in the final manuscript. We recognize that the teaser of Figure 2 may be confusing, and we will carefully redesign it for better understanding. \n\n\n[1] Wang, Haiyang, et al. \"Cagroup3d: Class-aware grouping for 3d object detection on point clouds.\" Advances in Neural Information Processing Systems 35 (2022): 29975-29988."}, {"review_id": "J3xs13DcBg", "reviewer": "Reviewer_MtnQ", "summary": "The paper proposes a pre-training strategy to distill the knowledge for large image foundation models. The method proposed is based on a mask auto-encoder transformer model and the setup contains several modules or losses. First, the masking strategy is guided by segmentation labels obtained using foundation models. Moreover, two additional losses at the scene and object level are incorporated to distill information from image foundation models. Lastly, a loss is used to align features from the point encoder to a text encoder.", "questions": "I do not have any questions for the authors. I believe the paper needs to significantly increase the evaluation to show which is the real contribution wrt previous work and to support the claims of the paper.", "limitations": "The paper does not analyze properly the limitations of the method. For example, authors could discuss: some of the foundation models used are trained with annotated segmentation masks. Does this pose a problem for the test sets used on the paper? how similar are the images used for training these foundation models to the images used during pre-training and downstream tasks?", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper proposes a combination of several foundation models with different modalities, such as segmentation masks, unsupervised pre-trained features, and text embeddings. Moreover, the paper is easy to follow.", "weaknesses": "- The novelty of the methods is questionable since other methods have used text foundation models (Language-Grounded Indoor 3D Semantic Segmentation in the Wild), and image pre-trained models (OpenScene: 3D Scene Understanding with Open Vocabularies), or both (ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding). The paper proposes to combine similar ideas, but they are not new and not enough comparisons to those other pre-training strategies are provided.\n\n- To use an informed mask guidance has been proposed before and the paper misses to mention all related research in this area. Moreover, the proposed masking method is not particular to 3D data and these other strategies for images could be used instead. These papers should be cited and compared to these approaches.\n\tSemMAE: Semantic-Guided Masking for Learning Masked Autoencoders\n\tWhat to Hide from Your Students: Attention-Guided Masked Image Modeling\n\n- The evaluation in general is limited and the only conclusion the reader can draw from these results is that Bridge3D is a better pre-training strategy than vanilla Point-MAE (which is not surprising), and PiMAE. The other comparisons provided in the table are comparing apples and oranges. Most of the methods in Table 1 use VoteNet as an object detection framework, a much older framework than the ones used with Bridge3D. Is the improvement coming from the pre-training or from the newer detection frameworks? Moreover, some of the numbers in the table are not correct. For example, DepthContrast provides results for two object detection frameworks and architectures, VoteNet and H3DNet. However, only the results for VoteNet are presented which are much lower than H3DNet (64.0/42.9/61.6/35.5 vs 69.0/50.0/63.5/43.4). Moreover, these methods use different architectures which makes the pre-training strategies not comparable. In order to claim superiority over all these pre-training strategies the same model and framework for object detection should be used. Since, the proposed method is based on MAE and therefore limits the architecture choice, at least the performance of the proposed method with a VoteNet framework (and other methods with the newer object detection frameworks) should be reported.\n\nRegarding semantic segmentation, the evaluation has also limitations. First, only results for S3DIS are provided. This data set is small and relatively large models overfit very easily, most of the recent improvements in the data set are obtained by the type of data augmentations used. I would suggest the authors to report results on a large data set such as ScanNet, which, although not perfect, is more representative of this task. Moreover, there are other works missing in Table 2 that appear in Table 1, for example [27]. They report 72.2 mIoU which is higher than the one obtained with the proposed method, Bridge3D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "With gratitude for the reviewer's valuable insights, we will make appropriate revisions to our paper. In the following section, we address each of the reviewer's questions and comments in detail.\n\n**W1: The novelty of this method**\n\n**1, Difference to the Language-Grounded Indoor 3D Semantic Segmentation in the Wild and OpenScene**. Unlike those previous works that focus solely on distilling either text to 3D or images to 3D, our method innovatively integrates both 2D visual information and textual descriptions to enrich 3D representations. In this paper, we have compared with a very similar and recent CVPR 2023 paper, CLIP2Scene [9] in relative work and ablation study sections. CLIP2Scene paper explores the transfer of CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network, which aligns closely with our focus. We will also discuss those two papers in relative works in the final version. \n\n**2, Differnce to the ULIP**.\nThe ULIP method is specifically tailored for pre-training on instance-level datasets such as ModelNet40 and ScanObjectNN and is hard to be directly applied to 3D scene understanding tasks. Furthermore, our method overcomes the limitations of ULIP that need text labels to generate text features. We leverage powerful image captioning models to generate scene text descriptions and utilize CHATGPT to refine these texts without requiring annotations. \n\n**3, Novelty**.\nTo the best of our knowledge, our work first harnesses multiple foundation models for self-supervised 3D scene understanding. The novelty of our approach can be summarized into three key areas: (1) We introduce a pioneering masking and patch-dropping strategy founded on foundation models. Unlike previous 3D masking methods that were confined to instance-level 3D objects, our approach breaks new ground by extending its applicability to 3D scene understanding. (2) We uniquely explore the utilization of highly accurate masks, extracted from state-of-the-art foundation models like SAM and Grounding Dino, a direction not previously pursued in existing works. (3) Our research crafts a novel scene-level and object-level multi-modality knowledge distillation method. This method is innovative in its approach to pre-training a 3D network, drawing upon features, semantic masks, and captions derived from foundation models, forging a new path in the field of 3D representation learning.\n\n**W2: Unreferenced informed mask guidance methods** \n\nThank you for highlighting previous work in informed mask guidance such as SemMAE and What to Hide from Your Students. We will cite those two papers in the revised version. In our work, we have compared our work with a\nCVPR 2023 paper I2P-MAE [63] in relative works. The I2P-MAE employs 2D semantic saliency maps from CLIP to guide 3D masking strategies. This approach shares commonalities with SemMAE and \"What to Hide from Your Students\" but diverges when applied to our context. SemMAE, What to Hide from Your Students, and I2P-MAE leverage self-attention of class tokens to obtain masks in object-centric data. However, our research is designed for scenes that contain multiple objects, making those self-attention-based methods inapplicable. This differentiation indicates the novelty of our approach.\n\n**W3: The limitation of evaluation**\n\nDue to the character limitation, we put this part in the global response.\n\n**W4: 3D segmentation results in ScanNet**\n\nRecent plain transformer-based works such as Pixel4Point[47] and ACT[1] only evaluate their results in the S3DIS dataset. However, we acknowledge your concerns regarding the S3DIS dataset's limitations. To evaluate the versatility of our approach, we evaluate our method in  ScanNet segmentation tasks.  Table 1 demonstrates that Bridge3D achieves SOTA performance in segmentation tasks on ScanNet even starting from a comparatively lower baseline. It should be notice that SR-UNet is the backbone designed specifically for segmentation, but the plain transformer is a more general backbone that can be applied in both detection and segmentation.\n\n**Table 1: 3D semantic segmentation results on ScanNet dataset.** \n\n| Methods          | Pre-trained  | $mIoU$                  |\n|------------------|--------------|-------------------------|\n| SR-UNet [58]        | *None*       | 72.2                    |\n| DepthContrast [65]    | ✓            | 73.1                    |\n| Hou et al. [27]      | ✓            | 73.8                    |\n| PointContrast [58]   | ✓            | **74.1(+1.9)**          |\n| Ours(from scratch)| *None*       | 67.3                    |\n| +Bridge3D        | ✓            | **73.9(+6.6)**          |\n\n**W5: Missing works**\n\nThank you for bringing our attention to the discrepancy between Tables 1 and 2, specifically with regard to work [27]. The absence of [27] was an unintentional omission, and we regret any confusion this may have caused. It is important to note that the mIoU of 72.2 reported by [27] is utilizing a more powerful baseline. The improvement we achieve with our method over the transformer baseline is substantially greater compared to other pre-training methods.\n\n****W6: Limitations of the paper are not discussed****\n\nWe also put this part to the global response part.\n\nAs for the question about foundation models, all foundation models used in our method do not leverage data from downstream tasks for training. This ensures an unbiased approach to downstream tasks. Your inquiry into the data similarities among foundation models and downstream tasks is a thoughtful and pertinent one. However, it is hard to evaluate their similarities due to the large scale of data used for foundation model training. \n\n[1] Dong, Runpei, et al. \"Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?.\" The Eleventh International Conference on Learning Representations. 2022.\n\n\n\n\n\n"}]

Author rebuttal:
We appreciate the time and effort spent reviewing our submission. In this response, we address the major concerns raised by the reviewers and respond to other concerns in individual rebuttal responses.

**Reviewer XLcq, MtnQ: The limitations of the paper not discussed**

Thank you for your thoughtful suggestion. The following part is about limitations and future work.

While our work has achieved promising results in 3D detection and segmentation tasks, it encounters specific limitations that merit further exploration. A significant constraint is the method's reliance on existing foundation models for mask and caption generation, leading to several offline computations of intermediate results and time-consuming data preprocessing. This dependency also means that changing or updating these underlying models might affect overall performance. In the future, we aim to investigate the integration of high-speed models, such as mobile SAM, and create a more balanced and efficient system that can maintain high performance without being heavily tied to the pre-existing models currently in use. 

**Reviewer MtnQ: The Limitation of evaluation**

Thank you for your detailed insights into the evaluation section of our paper. We admit that the plain transformer-based compared method is not enough in the main paper. In Table 1, we add more transformer-based pre-training methods in 3D object detection tasks. Currently, due to the difference in backbones, transformer-based pretraining methods in 3D classification, detection, and segmentation (e.g. ACT [4], PiMAE [8], and MaskPoint [3]) only compare to other transformer-based methods. Our updated analysis confirms that Bridge3D outperforms existing transformer-based 3D pretraining methods, reinforcing our main conclusions. 

**Table 1: 3D object detection results on ScanNet and SUN RGB-D dataset based on 3DETR.**

| Methods                 | Pre-trained | SUN RGB-D $AP_{25}$ | SUN RGB-D $AP_{50}$ | ScanNetV2 $AP_{25}$ | ScanNetV2 $AP_{50}$ |
|-------------------------|-------------|---------------------|---------------------|---------------------|---------------------|
| 3DETR                   | *None*      | 58.0                | 30.3                | 62.1                | 37.9                |
| +Bridge3D(from scratch) | *None*      | 57.6                | 31.9                | 61.1                | 38.6                |
| +Point-Bert[1]          | ✓           | -                   | -                   | 61.0                | 38.3                |
| +Point-MAE [44]         | ✓           | 59.2                | 33.2                | 62.3                | 39.9                |
| +Point-M2AE [2]         | ✓           | -                   | -                   | 63.7                | 40.8                |
| +MaskPoint [3]          | ✓           | -                   | -                   | 63.4                | 40.6                |
| +PiMAE [8]              | ✓           | 59.9                | 33.7                | 63.0                | 40.2                |
| +ACT [4]                | ✓           | -                   | -                   | 63.5                | 41.0                |
| +Bridge3D               | ✓           | **62.5(+4.9)**      | **36.8(+4.9)**      | **66.3(+5.4)**      | **45.5(+6.2)**      |



However, we agree that our paper could benefit from a more comprehensive comparison. We applied our method to the VoteNet framework by replacing the PointNet++ architecture with the plain transformer architecture. The result in Table 2 from this adaptation demonstrates that Bridge3D exhibits superior performance.

**Table 2: 3D object detection results on ScanNet and SUN RGB-D dataset based on VoteNet.**

| Methods                 | Pre-trained | SUN RGB-D $AP_{25}$ | SUN RGB-D $AP_{50}$ | ScanNetV2 $AP_{25}$ | ScanNetV2 $AP_{50}$ |
|-------------------------|-------------|---------------------|---------------------|---------------------|---------------------|
| VoteNet                 | *None*      | 57.7                | 32.9                | 58.6                | 33.5                |
| +Bridge3D(from scratch) | *None*      | 59.6                | 34.9                | 60.8                | 37.3                |
| +PointContrast [28]     | ✓           | 57.5                | 34.5                | 59.2                | 38.0                |
| +Hou et al. [27]        | ✓           | -                   | 36.4                | -                   | 39.3                |
| +4DContrast [10]        | ✓           | -                   | 38.2                | -                   | 40.0                |
| +DepthContrast [65]     | ✓           | 61.6                | 35.5                | 64.0                | 42.9                |
| +DPCo [36]              | ✓           | 60.2                | 35.5                | 64.2                | 41.5                |
| +Bridge3D               | ✓           | **63.7(+4.1)**      | **39.5(+4.6)**      | **67.1(+6.3)**      | **44.3(+7.0)**      |







[1] Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., & Lu, J. (2022). Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[2] Zhang, Renrui, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. "Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training." Advances in neural information processing systems 35 (2022)

[3] Liu, H., Cai, M., & Lee, Y. J. (2022, October). Masked discrimination for self-supervised learning on point clouds. In European Conference on Computer Vision. Cham: Springer Nature Switzerland.

[4] Dong, Runpei, et al. "Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?." The Eleventh International Conference on Learning Representations. 2022.

Meta-review: In this paper, the authors propose Bridge3D, a method for 3D scene representation learning by leveraging recent foundational models, such as segmentation and captioning models. After the rebuttal, all reviewers found their concerns addressed and recommend acceptance. After reading all reviews and rebuttals, the AC agrees with the decision.
Decision: Accept (poster)