SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Network architectures. For the 3D backbone encoder, we utilize the same architecture as Point-MAE [50]. For the image branch, we follow DINOV2 ViT-B [49] to divide  $518 \times 518$  images into regular patches with a size of  $37 \times 37$ , before the ViT backbone. For the image branch, we directly utilize the CLIP ViT-B [54] to extract text features. For image captioning, we leverage Tag2text [34].
Pre-training. During this stage, we perform training of the model for 120 epochs by employing the ScanNet dataset [14] consisting of point clouds and their corresponding images. For the text prompts, we only utilize all class names of ScanNet and SUN RGB-D as the prompts and filter other classes generated from image captioning. We use AdamW [45] optimizer with a base learning rate of 5e-4 and weight decay of 5e-2, along with a batch size of 64. The whole masking ratio  $r_w$  is set to 70% and the drop ratio  $r_d$  is set to 40%. The cosine learning rate scheduler is applied, with a drop path rate and warm-up epochs set to 0.1 and 10, respectively. The encoder depth is set to 6, and we utilize the same decoder as Point-MAE [50], with the decoder depth set to 2.
Methods	Pre-trained	SUN RGB-D	ScanNetV2
AP25	AP50	AP25	AP50
VoteNet [52]	None	57.7	32.9	58.6	33.5
PointContrast [65]	✓	57.5	34.5	59.2	38.0
Hou et al. [32]	✓	-	36.4	-	39.3
4DContrast [10]	✓	-	38.2	-	40.0
DepthContrast [72]	✓	61.6	35.5	64.0	42.9
DPCo [41]	✓	60.2	35.5	64.2	41.5
3DETR [47]	None	58.0	30.3	62.1	37.9
+Bridge3D(from scratch)	None	57.6	31.9	61.1	38.6
+Point-BERT[67]	-	-	-	61.0	38.3
+Point-MAE [50]	✓	-	-	63.4	40.6
+MaskPoint [42]	✓	-	-	63.4	40.6
+ACT [16]	✓	-	-	63.5	41.0
+PiMAE [8]	✓	59.9	33.7	63.0	40.2
+Bridge3D	✓	61.8(+3.8)	35.9(+5.6)	65.3(+3.2)	44.2(+6.3)
GroupFree3D [44]	None	63.0	45.2	67.3	48.9
+Bridge3D(from scratch)	None	62.2	45.0	66.1	48.3
+Point-MAE [50]	✓	63.9	46.1	67.4	49.8
+PiMAE [8]	✓	65.0	46.8	67.9	50.5
+Bridge3D	✓	67.9(+4.9)	48.5(+3.3)	69.1(+1.8)	51.9(+3.0)
Table 1: 3D object detection results on ScanNet and SUN RGB-D dataset. We adopt the average precision with 3D IoU thresholds of 0.25 ( $AP_{25}$ ) and 0.5 ( $AP_{50}$ ) for the evaluation metrics.
Methods	S3DIS	ScanNetV2
Pre-trained	mIoU	mAcc	mIoU	mAcc
SR-UNet [65]	None	68.2	75.5	72.1	80.7
PointContrast [65]	✓	70.9	77.0	74.1	81.6
DepthContrast [72]	✓	70.6	-	73.1	-
Hou et al. [32]	✓	72.2	-	73.8	-
Standard Transformer [67]	None	60.0	68.6	-	-
PointBert [67]	✓	60.8	69.9	-	-
PViT [53]	None	64.4	69.9	-	-
PViT+Pix4Point [53]	✓	69.6	75.2	-	-
Ours(from scratch)	None	61.1	67.2	67.3	73.1
+Point-MAE [50]	✓	64.8	70.2	-	-
+Bridge3D	✓	70.2 (+9.1)	76.1(+8.9)	73.9(+6.6)	80.2(+7.1)
Table 2: 3D semantic segmentation results on S3DIS dataset. We adopt the mean accuracy (mAcc) and mean IoU (mIoU) for the evaluation metrics.
4.2 Results on Downstream Tasks
For fine-tuning downstream tasks, we discard decoders in pre-training and append task-specific decoders onto the encoder for different tasks.
Object Detection. To demonstrate the generality of the proposed method, we also pre-train it on the indoor ScanNetV2 dataset [14] and subsequently fine-tune our method on the object detection task in ScanNetV2 dataset and SUN RGBD [75]. We report our performance on indoor 3D detection based on SOTA methods 3DETR [47] and GroupFree3D [44]. The Table 1 indicates that Our method achieves  $66.3\mathrm{AP}_{25}$ $(+4.2)$  and  $45.5\mathrm{AP}_{50}$ $(+7.6)$  compared to the baseline 3DETR on the ScanNetV2 dataset and also brings significant improvements to both models, surpassing previous baselines consistently in all other datasets and criteria. These experiments' results showcase our method's
Point-MAE	Semantic Guided Reconstruction	Scene-level Distillation	Object-level Distillation	ScanNetV2	S3DIS
AP25	AP50	mIoU	mAcc
✓	62.3	39.9	64.8	70.2
✓	✓	63.2	41.1	66.2	71.1
✓	✓	✓	64.4	43.0	68.4	73.7
✓	✓	✓	✓	65.3	44.2	70.2	76.1
Table 3: The effectiveness of each component. Ablation study on the effectiveness of each component on 3D object detection and semantic segmentation tasks.
Text	Image	ScanNetV2	S3DIS
AP25	AP50	mIoU	mAcc
62.1	37.9	61.1	67.2
✓	64.2	42.5	67.8	74.1
✓	64.7	43.3	68.3	74.5
✓	✓	65.3	44.2	70.2	76.1
Table 4: The effectiveness of each modality. Ablation study on the effectiveness of each modality on 3D object detection and semantic segmentation tasks.
effectiveness in learning superior 3D representations for object detection, highlighting its potential to benefit a wide range of 3D applications.
Semantic Segmentation. In Tab. 2, we present the semantic segmentation results on the S3DIS dataset. Despite their efforts, prior 3D self-supervised methods such as PointContrast[65] and [72] only achieved marginal improvements post-pre-training (+2.7 and +2.4 in  $mIoU$ ). Conversely, Pix4Point [53], utilizing a pre-trained 2D model, demonstrated significant progress compared to training from scratch. Most notably, our proposed method incorporates multiple foundation models during pre-training, elevating the metrics by 10.0 and 10.3 respectively, markedly surpassing other state-of-the-art 3D self-supervised methods. These results substantiate the effectiveness of utilizing multiple foundation models for enhancing 3D representation learning for semantic segmentation.
The effectiveness of Each Component. As shown in Table 3, the results indicate that each component, including the foreground-aware masking strategy, multi-modal scene-level knowledge distillation, and multi-modal object-level knowledge distillation, contributes to better results. Moreover, when we combined all components, our proposed method achieved the best performance. The foreground-aware masking strategy proved to be important as it enhanced the learning of foreground object representations in the 3D masked autoencoder. The multi-modal scene-level knowledge distillation, which leverages an image captioning model to generate text descriptions from paired images of point clouds, helped bridge the gap between 3D and text at the scene level. The multi-modal object-level knowledge distillation, which uses foundation models to generate accurate object-level masks and semantic text information, bridged the gap between 3D, 2D, and text object-level features. Overall, our ablation study demonstrates the effectiveness of each component in our proposed framework and highlights the importance of leveraging foundation models to improve 3D scene representation learning.
The effectiveness of Each Modality. Table 4 provides a clear insight into the significant contribution of each modality to the overall performance of our method. By integrating all modalities, our method realizes its full potential, showcasing the best results. However, it's worth noting the inherent resilience our method exhibits to modality variations. This adaptability implies that even when the modality mix is altered or reduced, the system remains relatively unaffected in terms of its output quality. This inherent resilience not only underscores the robust architecture of our model but also offers users the freedom to customize the framework based on their specific requirements. The adaptable nature of modality inclusion thus ensures that our method remains both versatile and efficient, enabling users to balance computational overhead with optimal performance.
Semantic Mask Method	ScanNetV2	S3DIS
AP25	AP50	mIoU	mAcc
Point-MAE [50]	62.3	39.9	64.8	70.2
MaskCLIP [76] + Superpixels [1]	64.1	43.0	67.1	73.4
Bridge3D	65.3	44.2	70.2	76.1
Table 5: The effectiveness of mask. Ablation study on unsupervised mask methods on 3D object detection and semantic segmentation tasks.
Methods	Pre-trained	SUN RGB-D	ScanNetV2
AP25	AP50	AP25	AP50
CAGroup3D [63]	None	66.8	50.2	75.1	61.3
+Bridge3D (scene & object level distillation)	✓	68.7	52.1	76.3	62.2
Table 6: The performance on SOTA method. 3D object detection results on ScanNet and SUN RGB-D dataset based on CAGroup3D.
The Effectiveness of Masks from Foundation Models. In Table 5, we conduct a comparative study of mask generation strategies employed by our method and those utilized by prior works. For instance, CLIP2Scene[9] employs MaskCLIP [76] to produce semantic masks. However, this approach fails to provide instance segmentation masks and is solely capable of guiding text-to-3D knowledge distillation. Alternatively, SLidR [59] leverages superpixels [1] for generating object-level masks. Despite this, the superpixels lack semantic information, limiting their use to guiding 2D-to-3D knowledge distillation. In contrast, our method generates instance semantic information, enhancing the functionality and accuracy of the masks. For a comparative analysis, we combine superpixels and MaskCLIP to direct both 2D-to-3D and text-to-3D distillation, essentially mimicking the fusion of CLIP2scene and SLidR. The experimental results reveal that masks generated through our method, which leverages foundation models, yield substantial improvements compared to the combined use of MaskCLIP and superpixels.
Apply Bridge3D in SOTA Method. Our method's adaptability extends to its successful application to the recent state-of-the-art (SOTA) work, CAGroup3D [63]. While the network structure made a direct application of the plain transformer challenging, we devised a unique adaptation, pre-training the CAGroup3D's backbone with our scene and object-level distillation, excluding the reconstruction part. Table 2 illustrates how our pre-training approach can enhance and benefit current SOTA 3D detection methodologies, showcasing the potential reach of our technique. Importantly, it should be noted that our framework is optimized for plain transformer-based backbones, and thus the application of Bridge3D to alternative backbones may lead to a reduction in performance.
In this work, we introduce a pioneering method Bridge3D that capitalizes on foundation models to overcome the hurdles in self-supervised 3D learning. This innovative approach not only enables more focused learning on foreground object representation but crucially bridges the domain gap between 3D and text at the scene level. Furthermore, it enriches the quality of 3D scene representation learning by generating highly accurate object-level masks and semantic textual information, effectively bridging the gap between 3D, 2D, and text object-level features. Our comprehensive experimental results corroborate the superior effectiveness of our approach in amplifying 3D scene understanding. However, the current work primarily focuses on indoor 3D scene understanding, which constitutes a limitation. Looking ahead, we plan to broaden the applicability of Bridge3D to encompass a more diverse set of 3D tasks, including outdoor scene understanding and open-vocabulary 3D tasks. Our work is expected to have no negative societal implications.