SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhimin Chen
Clemson University zhiminc@clemson.edu
Longlong Jing
The City University of New York  
ljing@gradcenter.cuny.edu
Yingwei Li3
Johns Hopkins University  
yingwei.li@jhu.edu
Bing Li $^{\text{※1}}$
Clemson University bli4@clemson.edu
Foundation models have achieved remarkable results in 2D and language tasks like image segmentation, object detection, and visual-language understanding. However, their potential to enrich 3D scene representation learning is largely untapped due to the existence of the domain gap. In this work, we propose an innovative methodology called Bridge3D to address this gap by pre-training 3D models using features, semantic masks, and captions sourced from foundation models. Specifically, our method employs semantic masks from foundation models to guide the masking and reconstruction process for the masked autoencoder, enabling more focused attention on foreground representations. Moreover, we bridge the 3D-text gap at the scene level using image captioning foundation models, thereby facilitating scene-level knowledge distillation. We further extend this bridging effort by introducing an innovative object-level knowledge distillation method that harnesses highly accurate object-level masks and semantic text data from foundation models. Our methodology significantly surpasses the performance of existing state-of-the-art methods in 3D object detection and semantic segmentation tasks. For instance, on the ScanNet dataset, Bridge3D improves the baseline by a notable margin of  $6.3\%$ . Code will be available at: https://github.com/Zhimin-C/Bridge3D
In recent years, task-agnostic pre-trained representations have fundamentally reshaped the landscape of Natural Language Processing (NLP), driven by the success of foundation models such as GPT-3 [55], PALM [13], T-NLG [31], and BERT [7]. Parallel advancements have been observed in the realm of computer vision, where foundation models like CLIP [54], Grounding DINO [43], DINOV2 [49], BLIP [39], and SAM [37] have established new benchmarks in 2D vision tasks, emerging as the leading approach for achieving state-of-the-art performance. However, the potential of these powerful models in advancing 3D scene understanding is yet to be fully realized, primarily due to the limited availability of large-scale 3D-text pair datasets and the considerable cost associated with procuring high-quality 3D annotations. Despite recent studies demonstrating the potential of individual foundation models like CLIP [54] or MOCO [27] in enhancing 3D scene understanding, a comprehensive exploration of the utility of other foundation models and their synergistic combinations remains a largely uncharted territory.
To address this challenge, we propose a novel framework Bridge3D that harnesses the strengths of multiple foundation models to advance 3D representation learning through a self-supervised learning
Figure 1: The motivation of Bridge3D. The driving force behind Bridge3D is to create a method that bridges the gap between 3D models and text/image foundation models via self-supervised learning.
approach. Specifically, the Bridge3D leverages image captioning outputs from foundation models to generate 3D scene prompts, establishing a bridge between 3D and text domains for scene-level knowledge distillation. These generated prompts are further utilized to produce instance segmentation results with the assistance of Grounding DINO and SAM. Subsequently, we employ a 3D network trained on a self-supervised task that distillates the knowledge from text and 2D to point clouds at the scene level. This is followed by the distillation of these multidimensional features into 3D features at the object level. Additionally, to optimize point reconstruction, we propose an inventive masking and patch-dropping strategy that redirects the model's attention toward foreground object representation learning.
Our proposed framework effectively circumvents three significant hurdles in 3D self-supervised learning. Firstly, the adoption of foundation models in our novel masking and dropping strategy allows the network to concentrate more on foreground object representation learning, thereby enhancing the performance of the 3D model. This is a distinct shift from traditional 3D masked autoencoder methods that rely on a random masking strategy and reconstruct all point clouds, which impairs representation learning due to the imbalance between foreground and background points. Secondly, the lack of datasets incorporating both 3D and text description pairs significantly hampers the potential for large-language models to contribute to 3D understanding. To overcome this obstacle, Our method first employs image captioning to generate text descriptions from paired images of point clouds, effectively bridging the 3D-text gap at the scene level. This novel integration of 3D and text modalities presents a compelling new frontier for improving self-supervised 3D scene understanding. Lastly, our approach stands in contrast to previous methodologies [9; 59] that facilitated either 2D to 3D or text to 3D distillation in isolation due to inherent limitations in mask generation. Instead, our strategy leverages foundation models to generate highly precise object-level masks and semantic text information. This approach seamlessly integrates object-level 3D, visual, and textual features, thereby significantly enhancing the quality of 3D scene representation learning.
We evaluate our method on multiple datasets, including SUN RGB-D [75] and ScanNet [14] for 3D object detection and S3DIS [5] for 3D semantic segmentation. Our approach outperforms state-of-the-art self-supervised learning methods in both tasks, demonstrating the effectiveness of our proposed framework. The contributions of Bridge3D can be summarized as follows:
1. We propose a novel masking and patch-dropping strategy based on foundation models to refine the focus of the network on foreground representation learning for 3D masked autoencoders.
2. We propose a novel scene-level and object-level multi-modality knowledge distillation method that pre-trains a 3D network via features, semantic masks, and captions obtained from foundation models
Figure 2: Overview of Bridge3D. Our method employs features, semantic masks, and captions derived from foundation models to improve 3D representation learning. We use semantic masks to guide the masking and reconstruction phases in the masked autoencoder, which intensifies the network's attention on foreground objects. At the scene level, we use image captioning foundation models to bridge the scene-level 3D-text gap. Additionally, we facilitate the distillation of well-learned 2D and text representations to the 3D model at the object level by leveraging foundation models to generate accurate object-level masks and semantic text information.
3. To the best of our knowledge, this is the first research to harness multiple foundation models for self-supervised 3D scene understanding. This pioneering approach has been shown to outperform state-of-the-art methods in various downstream tasks.
3D Self-Supervised Representation Learning. Recently, self-supervised pre-training on unlabeled point clouds [2; 24; 12; 66; 73; 35; 22] has shown promising transferable ability, providing a good network initialization for downstream fine-tuning. Several methods have been proposed for pre-training point cloud features, including learning relative position [58], multiple pretext tasks [26], and contrastive learning [17; 33; 57; 65; 72; 3; 35; 23; 21]. Info3D [57] extends the InfoMax and contrastive learning principles to 3D shapes. PointContrast [65] conducts point-level contrast on two transformed views of the same point cloud. Zhang [72] contrasts instance-level representations obtained from the same scenario but processed by different model architectures. CrossPoint [3] introduces an auxiliary multi-modal contrastive objective that captures 3D-2D correspondence, leveraging the complementary attributes of point clouds and images. Point-BERT [67] uses pre-trained tokenizers to indicate discrete point tokens, while Point-MAE [50] applies Masked Autoencoders (MAE) to directly reconstruct the 3D coordinates of masked tokens. Our proposed method uses Point-MAE as the baseline, but leverages foundation models to guide the masking and reconstruction stages. Additionally, we leverage image and text knowledge from foundation models to enhance 3D self-supervised learning.
Foundation Models. The field of AI research has experienced a paradigm shift with the emergence of models trained on massive amounts of data at scale, commonly referred to as foundation models [19; 4; 56; 62; 25; 60]. These models have demonstrated remarkable performance in various language and visual-related tasks. The use of large-scale text pre-training on attention-based models [15; 71] has led to the increasing popularity of vision-language models (VLM) due to their impressive performance in visual understanding tasks [54; 46; 40]. Recent advancements in contrastive learning have enabled CLIP [54] to perform multimodal learning with 400M data crawled from the web. CLIP has been extended for high-efficiency model training and cycle consistency through various methods
(a) Semantic masks of MaskCLIP with all labels as prompts.
(b) Semantic masks of Ground-SAM with all labels as prompts.
Figure 3: The comparison of zero-shot semantic results. (a) MaskCLIP failed to perform semantic segmentation accurately, yielding low accuracy overall (b) when all possible labels were used as prompts, the instance segmentation from foundation models results were prone to false positives (as seen in the Chair and Bathtub in this example) (c) by leveraging image captioning model to generate text prompts, the performance of the instance segmentation from Ground-SAM is further improved. This approach is beneficial for 3D scene understanding.
(c) Semantic masks of Ground-SAM with image caption prompts.
[40; 39; 38]. BLIP [39] includes text-to-image generation as an auxiliary task, which results in better performance by utilizing synthetic data as a bonus. More recently, the success of the foundation models has been achieved in the pure computer vision area. Segment Anything (SAM) [37] has been proposed to act as a generic image segmentation model trained on the large visual corpus. To overcome the drawback of CLIP which overlooks the visual local information, DINOV2 [49] is proposed, which is trained with self-supervised learning and achieves results that match or surpasses the standard approach used in task-specific fields. Grounding DINO [43] extends a closed-set detector DINO to open-set object detection by performing vision-language modality fusion at multiple phases.
Self-supervised 3D Understanding with Foundation Models. A number of studies have proposed strategies for knowledge transfer from pre-trained 2D foundation models to 3D representations at the object level [28; 30; 69; 68; 51]. For comprehensive scene understanding, recent efforts have improved 3D point representations by exploiting pixel-point alignments for distillation or contrastive learning [59; 9]. The I2P-MAE [70] approach takes advantage of 2D semantic saliency maps from CLIP [54] to guide masking and facilitate knowledge distillation from 2D to 3D at the instance level. Despite these advancements, most current 3D understanding methodologies employing foundation models focus predominantly on CLIP and distill knowledge through feature-level consistency or contrastive learning [9; 59]. The potential to utilize the capabilities of other foundation models, such as BLIP [39] for image-to-text captioning, SAM [37] for mask generation, Grounding DINO [43] for zero-shot detection, and DINOV2 [49] for high-performance features with detailed localized information, remains largely unexplored. Therefore, we propose to advance self-supervised 3D scene understanding by newly incorporating features, semantic masks, and captions obtained from various foundation models, achieving superior performance compared to state-of-the-art methodologies.
The pipeline of Bridge3D is illustrated in Fig. 2. Our approach employs semantic information and features extracted from well-established foundation models to enhance 3D scene representation learning using self-supervised learning. The proposed method consists of three components: a semantic-guided masked autoencoder, multi-modal scene-level knowledge distillation, and multimodal object-level knowledge distillation.
3.1 Mask Generation by Foundation Models
Our proposed methodology leverages existing foundation models to produce instance segmentation masks. Initially, we use Tag2text [34], based on BLIP [39], to create image captions. We leverage ChatGPT to filter captions objects neither in ScanNet [14] nor SUN RGB-D [61] dataset to generate text prompts for the 3D scene. Subsequently, we employ SAM [37] to generate masks for the image.
(a) Masking results of SLIC super-pixels.
(b) Masking results by our method which combines Grounding-DINO and SAM.
Figure 4: The comparison of masks. (a) Masks generated from super-pixel and (b) Masks generated from our method. Our method can provide much more accurate object-level masks compared to super-pixel and thus benefits the multi-modality knowledge distillation.
Lastly, we use the text labels as prompts for Grounding Dino [43] to create the corresponding bounding box, which is then inputted into SAM to produce both zero-shot instance segmentation labels and segmentation masks  $\mathcal{O}_1,\ldots ,\mathcal{O}_N$ . To establish the dense visual token-point token correspondence  $x_{i},p_{i}$ , we calibrate the point cloud with the respective images, where  $x_{i}$  and  $p_i$  signify the  $i$  paired image feature and point feature. This procedure is completed offline and saved locally, with the generated labels being used directly during the self-supervised training stage. As shown in Fig.3 and Fig.4, the instance segmentation masks generated from foundation models outperform previous methods in terms of semantic results and object-level masks. Furthermore, Fig. 3 demonstrates that the performance of the foundation model is further improved when using caption methods as prompts and filtering out 3D-unrelated text.
3.2 Semantic Guided 3D Masked Autoencoder
To let the 3D model understand 3D-specific representations, we leverage Point-MAE [50] as the baseline for pre-training, which learns a meaningful representation with the pretext task of recovering the original inputs from visible ones. The Point-MAE [50] method utilizes standard Transformers as the backbone of its architecture, with an encoder-decoder structure that is asymmetric. The encoder takes visible tokens  $T^v$  as input and generates encoded tokens  $T^e$ , while the decoder contains fewer Transformer blocks and is responsible for generating the reconstructed masked 3D coordinates. Positional embeddings are incorporated into each Transformer block to provide location-based information. The encoded tokens  $T^e$  are padded with learnable mask tokens  $T^m$  and sent to the decoder. A set of positional embeddings is added to each Transformer block in the decoder to provide location information to all tokens. The output of the decoder  $H^m$  is then passed through a simple fully connected layer to reconstruct the masked 3D coordinates  $P^{pre}$ . After that, it restores the coordinates of the points in each masked point patch, and to evaluate the accuracy of the predicted coordinates, it computes the reconstruction loss using  $l_2$  Chamfer Distance [18], which is formulated as:
$$
\mathcal {L} _ {m a e} = \frac {1}{M _ {m a s k}} \operatorname {C h a m f e r} \left(P ^ {p r e}, P ^ {m a s k}\right) \tag {1}
$$
where  $P_{mask}$  represents the ground truth of masked points.
Foreground-aware Masking and Patch Dropping. The Point-MAE [50] relies on a random masking strategy, resulting in dispersed attention across the entire image and insufficient focus on foreground objects. This dispersion of attention can lead to the model wasting computational resources on unimportant background elements, thereby leading to weaker learned representations. Moreover, the Point-MAE's approach to reconstructing all masked tokens, including those from the background, may further weaken representation learning as the model overly concentrates on the
background due to the foreground-background imbalance. Although MAE [70] has suggested using 2D semantic saliency maps to guide the masking of point tokens for 3D instances in classification tasks, the problem of effectively guiding 3D scene masking still remains a key research challenge.
To address those problems, we propose a semantic-guided masking strategy based on segmentation results obtained from foundation models. Specifically, for foreground objects obtained from the segmentation mentioned before, we mask a higher percentage  $r_f$  of foreground points compared to the whole masking ratio  $r_w$ . This generates a more challenging reconstruction task and forces the model to focus more on foreground objects. In addition, instead of inputting all patches  $\{P_i\}_{i=1}^M$  as in Point-MAE, we randomly drop a percentage  $r_d$  of background patches to obtain  $\{P_i\}_{i=1}^N$ . Where  $N = (1 - r_d) \times M$ . The transformer decoder reconstructs masked point patches by using features from visible tokens and the positional information of both visible patches and mask patches. The patches that are dropped will not be reconstructed by the decoder. With enough background patches dropped, the decoder sees fewer points to perform the trivial up-sampling, which further improves the 3D representation learning and accelerates the pre-training by reducing the input data.
3.3 Scene-level Multi-modal Knowledge Distillation
Although some works have investigated scene-level multimodal learning using image and point clouds [36; 11; 41; 70], exploring scene-level multimodal learning with text and point clouds remains a challenge due to the lack of corresponding text descriptions for current 3D scene datasets. To address this problem, we propose to leverage image captioning methods to generate corresponding captions and filter out other 3D irrelevant objects to obtain 3D scene description texts  $t_s$ . Then, we train the 3D network to align 3D point clouds with their corresponding scene-level images  $i_s$  and texts  $t_s$ . We formulate the proposed method below. Consider a set of  $N$  point cloud-image-text pairs  $\{F^{3D}, F^{2D}, F^{text}\}$ , where  $F^{3D}$  represents scene-level point cloud features from the encoder,  $F^{2D}$  is the corresponding image features obtained from pre-trained foundation model based on  $i_s$ , and  $F^{text}$  is the text features from pre-trained foundation model based on  $t_s$ . Our model maps scene-level 3D features  $F^{3D}$  to the hidden representation  $\hat{F}^m$  for each modality  $m$  with a projection head  $E_m$ . The mapping process can be formulated as:
$$
\hat {F} ^ {m} = E _ {m} \left(F ^ {3 D}\right), \tag {2}
$$
Due to the attributes to avoid representation collapsing, previous methods [9; 59] utilize InfoNCE loss [48] to conduct multi-modality knowledge distillation. However, in this work, we find that leveraging positive only  $L_{1}$  smooth loss generates better results. We think this is because those foundation models have learned discriminative features during the pre-training stage, and thus negative pairs are not necessary for the distillation stage. The scene-level distillation between 3D-image features; and 3D-text features are defined by:
$$
\mathcal {L} _ {\text {s c e n e}} = L _ {1} \left(\hat {F} ^ {2 D}, F ^ {2 D}\right) + L _ {1} \left(\hat {F} ^ {\text {t e x t}}, F ^ {\text {t e x t}}\right) \tag {3}
$$
where  $L_{1}$  represents the  $L_{1}$  smooth loss.
3.4 Object-level Multi-modal Knowledge Distillation
While the value of object-wise feature representations in downstream tasks like semantic segmentation and detection is well-proved [74; 29; 64; 6], the generation of unsupervised masks presents a substantial challenge. Traditional techniques in computer vision, such as Felzenszwalb-Huttenlocher [20] and super-pixel [1], have been employed in earlier methods [59]. However, these techniques yield subpar masking results and cannot generate semantic labels, which hinders their ability to bridge the 3D-2D-text gap and leaves the potential of powerful language foundation models unrealized. The recent CLIP2Scene method [9] uses the MaskClip [76] model to generate dense semantic predictions, but it falls short of generating instance semantic results, which prevents object-level visual representations from being distilled into 3D models. The inferior quality outputs of MaskClip, Felzenszwalb-Huttenlocher, and super-pixel impede 3D representation learning. In contrast, our proposed Bridge3D method leverages high-quality masks obtained from foundation models to guide object-level knowledge distillation, thereby enhancing 3D scene understanding.
In the object-level knowledge distillation phase of Bridge3D, we propose a distinctive approach to multi-modality distillation through reconstructed tokens from the decoder, as opposed to previous
methods [9; 59] that directly distill the knowledge from other modalities to 3D post-encoder. The Point-MAE uses the decoder to reconstruct these masked tokens and learn specific features. In our method, we not only task the decoder with reconstructing masked point clouds but also reconstruct text and visual features that correlate with visible point tokens to distill the multi-modal knowledge. We do not reconstruct text and visual features of masked tokens that are dropped in the point reconstruction part, as mentioned in Sec. 3.2. To be specific,  $I_{i}$  represents the visual features obtained from the pre-trained visual model and belonging to the mask  $O_{i}$ . We use the mapping function to group the points into corresponding masks:  $\mathcal{G}_1, \ldots, \mathcal{G}_k$ . For each scene, we compute mask visual features and mask point features by average pooling:
$$
f _ {l, i} ^ {2 D} = \frac {1}{\mathcal {O} _ {i}} \sum_ {i \in \mathcal {O} _ {j}} \left(I _ {j}\right) \tag {4}
$$
$$
\hat {f} _ {l, i} ^ {2 D} = \frac {1}{\mathcal {G} _ {i}} \sum_ {j \in \mathcal {G} _ {i}} \left(\mathcal {F} _ {2 D} \left(H _ {i}\right)\right) \tag {5}
$$
Where  $H_{i}$  represents visible tokens from the decoder,  $\mathcal{F}_{2D}$  is the projection head. For the text-to-point cloud knowledge distillation, as only foreground objects have the text information from the semantic labels, we choose corresponding foreground masks  $\mathcal{I}_1,\dots ,\mathcal{I}_s$  from  $\mathcal{G}$ . Text features are obtained following:
$$
f _ {l, i} ^ {\text {t e x t}} = \phi_ {\text {t e x t}} \left(t _ {s, i}\right) \tag {6}
$$
$$
\hat {f} _ {l, i} ^ {\text {t e x t}} = \frac {1}{\mathcal {J}} \sum_ {j \in \mathcal {J} _ {i}} \left(\mathcal {F} _ {\text {t e x t}} \left(H _ {j}\right)\right) \tag {7}
$$
Where  $t_{s,i}$  is the corresponding mask semantic label and  $\phi_{text}$  is the pre-trained text encoder, and  $\mathcal{F}_{text}$  is the projection head. We then transfer visual-text pairs to point-text pairs  $(f_{l,i}^{3D}, f_{l,i}^{text})$  and distill the knowledge from text to the point cloud in the object-level. The objective function is as follows:
$$
\mathcal {L} _ {\text {o b j e c t}} = \frac {1}{K} \sum_ {i} ^ {K} L _ {1} \left(\hat {f} _ {l, i} ^ {2 D}, f _ {l, i} ^ {2 D}\right) + \frac {1}{S} \sum_ {i} ^ {S} L _ {1} \left(\hat {f} _ {l, i} ^ {\text {t e x t}}, f _ {l, i} ^ {\text {t e x t}}\right) \tag {8}
$$
The  $L_{1}$  is the smooth  $L_{1}$  loss. Our final loss is the sum of previous loss terms.
$$
L _ {\text {f i n a l}} = L _ {\text {m a e}} + L _ {\text {s c e n e}} + L _ {\text {o b j e c t}} \tag {9}
$$
In this section, we first introduce the pre-training setting of Bridge3D. Then, we show the effectiveness of our method on several popular downstream tasks, including 3D object detection and 3D semantic segmentation. Finally, we conduct extensive ablation studies to show the effectiveness of each design. We put more details into the supplementary materials.
4.1 Self-supervised Pre-training