SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Runnan Chen<sup>1</sup> Youquan Liu<sup>2</sup> Lingdong Kong<sup>3</sup> Nenglun Chen<sup>1</sup> Xinge Zhu<sup>4</sup> Yuexin Ma<sup>5*</sup> Tongliang Liu<sup>6</sup> Wenping Wang<sup>7*</sup>
<sup>1</sup>The University of Hong Kong <sup>2</sup>Hochschule Bremerhaven  
<sup>3</sup>National University of Singapore <sup>4</sup>The Chinese University of Hong Kong  
<sup>5</sup>ShanghaiTech University <sup>6</sup>The University of Sydney <sup>7</sup>Texas A&M University
Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot performance on image classification and segmentation tasks. However, the incorporation of CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, we investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CLIP and further exacerbated during the propagation from the 2D to the 3D domain. To tackle these challenges, we propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-free semantic segmentation with  $28.4\%$  and  $33.5\%$  mIoU on ScanNet, improving  $4.7\%$  and  $7.9\%$ , respectively. For nuImages and nuScenes datasets, the performance is  $22.1\%$  and  $26.8\%$  with improvements of  $3.5\%$  and  $6.0\%$ , respectively. Code is available. $^{2}$
Scene understanding aims to recognize the semantic information of objects within their contextual environment, which is a fundamental task for autonomous driving, robot navigation, digital city, etc. Existing methods have achieved remarkable advancements in 2D and 3D scene understanding [1-10]. However, they heavily rely on extensive annotation efforts and often struggle to identify novel object categories that were not present in the training data. These limitations hinder their practical applicability in real-world scenarios where acquiring high-quality labelled data can be expensive and novel objects may appear [11-17]. Consequently, label-free scene understanding, which aims to perform semantic segmentation in real-world environments without requiring labelled data, emerges as a highly valuable yet relatively unexplored research topic.
Vision foundation models, e.g., Contrastive Vision-Language Pre-training (CLIP) [18] and Segment Anything (SAM) [19], have garnered significant attention due to their remarkable performance in addressing open-world vision tasks. CLIP, trained on a large-scale collection of freely available image-text pairs from websites, exhibits promising capabilities in open-vocabulary image classification. On
$^{2}$ https://github.com/runnanchen/Label-Free-Scene-Understanding.
Figure 1: We study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data. To accomplish this, we introduce a novel framework called Cross-modality Noisy Supervision (CNS). By effectively harnessing the strengths of CLIP and SAM, our approach simultaneously trains 2D and 3D networks, yielding remarkable performance.
the other hand, SAM learns from an extensive dataset comprising 1 billion masks across 11 million images, achieving impressive zero-shot image segmentation performance. However, initially designed for image classification, CLIP falls short in segmentation performance. Conversely, SAM excels in zero-shot image segmentation but lacks object semantics (Fig. 1). Additionally, both models are trained exclusively on 2D images without exposure to any 3D modal data. Given these considerations, a natural question arises: Can the combination of CLIP and SAM imbue both 2D and 3D networks with the ability to achieve label-free scene understanding in real-world open environments?
Despite recent efforts [20] leverage CLIP for image-based semantic segmentation, the pseudo labels generated by CLIP for individual pixels often exhibit significant noise, resulting in unsatisfactory performance. The up-to-date work [17] has extended this method to encompass 3D label-free scene understanding by transferring 2D knowledge to 3D space via projection. However, the pseudo labels assigned to 3D points are considerably noisier due to calibration errors, significantly limiting the accuracy of the networks. The primary challenge of utilizing vision foundation models for scene understanding is effectively supervising the networks using exceptionally noisy pseudo labels.
Inspired by SAM's impressive zero-shot segmentation capabilities, we propose a novel Cross-modality Noisy Supervision (CNS) framework incorporating CLIP and SAM to train 2D and 3D networks simultaneously. Specifically, we employ CLIP to densely pseudo-label 2D image pixels and transfer these labels to 3D points using the calibration matrix. Since pseudo-labels are extremely noisy, leading to unsatisfactory performance, we refine pseudo-labels using SAM's masks, producing more reliable pseudo-labels for supervision. To further mitigate error propagation and prevent the networks from overfitting the noisy labels, we consistently regularize the network predictions, i.e., co-training the 2D and 3D networks using the randomly switched pseudo labels, where the labels are from the prediction of 2D, 3D, and CLIP networks. Moreover, considering that individual objects are well distinguished in SAM feature space, we use the robust SAM feature to consistently regularize the latent space of 2D and 3D networks, which aids networks in producing semantic predictions with more precise boundaries and further reduces label noise. Note that the SAM feature space is frozen during training, thus severed as the anchor metric space for aligning the 2D and 3D features.
To verify the label-free scene understanding capability of our method in 2D and 3D real open worlds, we conduct experiments on indoor and outdoor datasets, i.e., ScanNet and nuScenes, where 2D and 3D data are simultaneously collected. Extensive results show that our method significantly outperforms state-of-the-art methods in understanding 2D and 3D scenes without training on any labelled data. Our 2D and 3D network achieves label-free semantic segmentation with  $28.4\%$  and  $33.5\%$  mIoU on ScanNet, improving  $4.7\%$  and  $7.9\%$ , respectively. For nuImages and nuScenes datasets, the performance is  $22.1\%$  and  $26.8\%$  with improvements of  $3.5\%$  and  $6.0\%$ , respectively. Quantitative and qualitative ablation studies also demonstrate the effectiveness of each module.
The key contributions of our work are summarized as follows.
- In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data.
- We propose a novel Cross-modality Noisy Supervision framework to effectively and synchronously train 2D and 3D networks with severe label noise, including prediction consistency and latent space consistency regularization schemes.
- Experiments conducted on indoor and outdoor datasets show that our method significantly outperforms state-of-the-art methods on 2D and 3D semantic segmentation tasks.
Scene Understanding. Scene understanding is a fundamental task for computer vision and plays a critical role for robotics, autonomous driving, smart city, etc. Significant advancements have been made by current supervised methods for 2D and 3D scene understanding [1-10, 21-34]. However, these methods heavily depend on extensive annotation efforts, which pose significant challenges when encountering novel object categories that were not included in the training data. In order to overcome these limitations, researchers have proposed self-supervised and semi-supervised methods [13, 14, 35-54] to train networks in a more data-efficient manner. Nevertheless, these approaches struggle to handle novel objects that are not present in training data and they often exhibit subpar performance when faced with significant domain gaps. Alternatively, some methods [11, 12, 55-64] focus on open-world scene understanding that identifies novel object categories absent in the training data. However, these methods still require expensive annotation efforts on the close-set category data. Other methods [56, 17, 65-67] try to alleviate 3D annotation efforts by distilling knowledge from 2D networks. However, they either need 2D labelled data to train networks or struggle to achieve satisfactory results in the context of label-free 3D scene understanding.
Recently, vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) [18] and Segment Anything (SAM) [19] have garnered significant attention due to their remarkable performance in open-world vision tasks. MaskCLIP [20] extends the capabilities of CLIP to image semantic segmentation, achieving promising results in 2D scene understanding without labelled data. Furthermore, CLIP2Scene [17] expands upon MaskCLIP to enable 3D scene understanding through the use of a 2D-3D calibration matrix. In this study, our focus lies in label-free understanding using vision foundation models. Our ultimate goal is to perform semantic segmentation in real-world environments without relying on labelled data. We hope that our work will inspire and motivate further research in this area.
Label-noise Representation Learning. Label-Noise Representation Learning (LNRL) aims to train neural networks robustly in the presence of noisy labels. Numerous methods have been developed to tackle this challenge. Some methods [68-74] focus on designing accurate estimators of the noise transition matrix, which establish a connection between the clean class posterior and the noisy class posterior. Others [75-81] devise noise-tolerant objective functions through explicit or implicit regularization, enabling the learning of robust classifiers on noisy data. Furthermore, Some approaches [82-86] delve into the dynamic optimization process of LNRL based on the memorization effects, i.e., deep models tend to initially fit easy (clean) patterns and gradually overfit complex (noisy) patterns. However, the above methods are insufficient when clean labels are unavailable, which makes them inadequate for addressing our problem with entirely noisy labels. Furthermore, these methods are primarily designed for a single modality, and the exploration of multiple modality scenarios is still an unexplored topic. In this paper, we study the LNRL across different modalities without any clean labels.
The vision foundation models have demonstrated remarkable performance in open-world scenarios. However, while CLIP excels in zero-shot image classification, it tends to produce noisy masks in segmentation tasks. Conversely, SAM generates impressive object masks in zero-shot segmentation but lacks object semantics. Therefore, our research aims to mitigate the impact of noisy labels by leveraging the strengths of various modalities, such as images, point clouds, CLIP, and SAM. In what follows, we revisit the vision foundation models applied in open-vocabulary classification and semantic segmentation, then present the Cross-modality Noisy Supervision (CNS) framework in detail.
Figure 2: Illustration of the overall framework. Specifically, we employ CLIP to densely pseudo-label 2D image pixels and transfer these labels to 3D points using the calibration matrix. Acknowledging that network predictions and pseudo-labels are extremely noisy, we refine pseudo-labels using SAM's masks for more reliable supervision. To mitigate error propagation and prevent the networks from overfitting the noisy labels, we randomly switch the different refined pseudo-labels to co-training the 2D and 3D networks. Besides, we consistently regularize the network's latent space using SAM's robust feature, which benefits the networks to produce clearer predictions.
3.1 Revisiting Vision Foundation Models
Vision foundation model refers to powerful models pre-trained on large-scale multi-modal data and can be applied to a wide range of downstream tasks [87]. CLIP and SAM are well recognized due to their impressive open-world image classification and segmentation performance.
CLIP combines an image encoder (ResNet [88] or ViT [89]) and a text encoder (Transformer [90]), which independently map image and text representations to a shared embedding space. CLIP trains both encoders using a contrastive loss on a dataset of 400 million image-text pairs sourced from the Internet that contain diverse image classes and textual concepts. This training approach enables CLIP to achieve impressive performance in open-vocabulary recognition. CLIP utilizes a two-step process for zero-shot image classification. Initially, it obtains image embeddings and generates text embeddings by incorporating the class name into a predefined template. Then, it compares the similarity of image and text embeddings to determine the class. SAM serves as the fundamental model for image segmentation. It undergoes training using over 1 billion masks on a dataset of 11 million licensed images, enabling it to adapt seamlessly to new image distributions and tasks without prior exposure. Extensive evaluations have demonstrated that SAM's zero-shot performance is remarkable, frequently rivaling or surpassing the results of previous fully supervised approaches. Note that SAM works impressively on zero-shot image segmentation but lacks object semantics.
MaskCLIP [20] extends the CLIP's capabilities to image semantic segmentation. Specifically, MaskCLIP modifies the attention pooling layer of CLIP's image encoder to enable pixel-level mask prediction instead of global image-level prediction. CLIP2Scene [17] further expands MaskCLIP into 3D scene understanding by transferring the pixel-level pseudo-labels from the 2D image to the 3D point cloud using a 2D-3D calibration matrix. However, their performance is unsatisfactory due to the significant noise in the pseudo labels. To this end, we leverage the strengths of vision foundation models to supervise networks under extremely noisy pseudo labels.
3.2 Cross-modality Noisy Supervision
As illustrated in Fig. 2, we introduce a novel Cross-modality Noisy Supervision (CNS) method that combines CLIP and SAM foundation models to concurrently train 2D and 3D networks for label-free scene understanding. Our approach involves pseudo-labelling the 2D images and 3D points using CLIP. To address the noise in pseudo-labelling, we employ SAM's masks to refine the pseudo-labels. To further mitigate error propagation and prevent overfitting to noisy labels, we
propose a prediction consistency regularization mechanism. This mechanism randomly switches refined pseudo-labels from different modalities to co-train the 2D and 3D networks. Additionally, we consistently regularize the network's latent space using SAM's robust feature, which aids in generating more accurate predictions. In the following sections, we provide a comprehensive overview of our method, along with detailed explanations and insights.
Pixel-point Projection. Since vision foundation models are pre-trained on 2D images, we transfer knowledge from 2D images to 3D point clouds via pixel-point correspondences. Note that the dataset already provides the camera poses, so we obtain the pixel-point correspondence  $\{c_i,s_i,x_i,p_i\}_{i = 1}^N$  via simple projection, where  $c_{i}$ ,  $s_i$  and  $x_{i}$  indicate the  $i$ -th paired image pixel feature by CLIP, SAM, and 2D network, respectively.  $p_i$  is the  $i$ -th paired point feature by the 3D network  $\mathcal{F}_{3D}$  and  $N$  is the number of pairs.
Pseudo-labeling by CLIP. We follow the pioneer methods, MaskCLIP and CLIP2Scene, that adopt CLIP to dense pseudo-label the 2D image pixels and transfer these labels to 3D points. Specifically, given the calibrated pixel-point features  $\{c_i,s_i,x_i,p_i\}_{i = 1}^N$ , we pseudo-label the pixel  $c_{i}$  as follows:
$$
P _ {c _ {i}} = \underset {l} {\arg \max } \psi \left(c _ {i} ^ {T}, t _ {l}\right), \tag {1}
$$
where  $c_{i}$  is the  $i$ -th pixel feature by CLIP's image encoder.  $t_{l}$  is the CLIP's text embedding of the  $l$ -th class name into a pre-defined template.  $\psi$  is the dot product operation.
Since pixel-point features  $\{c_i, s_i, x_i, p_i\}_{i=1}^N$  are spatial aligned, we can transfer the pseudo-label  $\{P_{c_i}\}_{i=1}^N$  to all image pixels and point coulds, i.e.,
$$
P _ {s _ {i}} = P _ {x _ {i}} = P _ {p _ {i}} = P _ {c _ {i}}, \tag {2}
$$
where  $\{P_{s_i}\}_{i = 1}^N,\{P_{x_i}\}_{i = 1}^N$  and  $\{P_{p_i}\}_{i = 1}^N$  are the transferred pseudo-labels from CLIP to SAM, 2D and 3D networks, respectively.
Label Refinement by SAM. Since CLIP is pre-trained on 2D images and text for zero-shot classification, it brings extreme noise to pixel-wise prediction. Nevertheless, another vision foundation model, Segment Anything (SAM), can provide impressive object masks in open-world environments, even rivaling or surpassing the previous fully supervised approaches. To this end, we consider refining the CLIP's pseudo-labels by SAM. Firstly, we follow the official implementation of SAM to generate all masks from the images. The mask ID of pixels are presented as  $\{M_{s_i}\}_{i = 1}^N$ , where  $M_{s_i}\in \{1,2,\dots,T\}$  and  $T$  indicates the number of masks. Once again, we could transfer the masks to all image pixels and point clouds based on the pixel-point correspondence  $\{c_i,s_i,x_i,p_i\}_{i = 1}^N$ , i.e.,
$$
M _ {c _ {i}} = M _ {x _ {i}} = M _ {p _ {i}} = M _ {s _ {i}}, \tag {3}
$$
where  $\{M_{c_i}\}_{i = 1}^N,\{M_{x_i}\}_{i = 1}^N$  and  $\{M_{p_i}\}_{i = 1}^N$  are the transferred masks from SAM to CLIP, 2D and 3D predictions, respectively.
We then adjust the pseudo-labels to be identical if they are within the same mask. For simplicity, we take the max voting strategy, i.e.,
$$
P _ {c _ {i}} ^ {*} = \arg \max  _ {l} \sum_ {c _ {j}, M _ {c _ {i}} = M _ {c _ {j}}} \mathbb {1} \left\{P _ {c _ {j}} = l \right\}, \tag {4}
$$
where  $\mathbb{1}\{\cdot\}$  is the indicator function.  $P_{c_i}^*$  is the SAM refined pseudo-label from  $P_{c_i}$ . To this end, all pixels and point pseudo-labels within the same mask will be corrected by the semantic label most pixels belonging to, i.e.,  $\{P_{c_i}^*\}_{i=1}^N$ ,  $\{P_{x_i}^*\}_{i=1}^N$  and  $\{P_{p_i}^*\}_{i=1}^N$ .
Prediction Consistency Regularization. After obtaining the refined pseudo labels of all image pixels and points  $\{P_{c_i}^*\}_{i = 1}^N$ ,  $\{P_{x_i}^*\}_{i = 1}^N$  and  $\{P_{p_i}^*\}_{i = 1}^N$ , we consider how to train both 2D and 3D networks. The training process consists of two stages. In the first stage, we simultaneously supervise the 2D and 3D networks by  $\{P_{x_i}^*\}_{i = 1}^N$  and  $\{P_{p_i}^*\}_{i = 1}^N$  with Cross-entropy loss:
$$
\mathcal {L} _ {s 1} = \sum_ {i} C E \left(\theta_ {s} \left(x _ {i}\right), P _ {x _ {i}} ^ {*}\right) + \sum_ {i} C E \left(\varphi_ {s} \left(p _ {i}\right), P _ {p _ {i}} ^ {*}\right), C E (x, y) = - \log \frac {\exp \left(\psi \left(x , t _ {y}\right)\right)}{\sum_ {l} \exp \left(\psi \left(x , t _ {l}\right)\right)}, \tag {5}
$$
where  $\theta_{s}$  and  $\varphi_{s}$  are linear mapping layers of 2D  $\{x_i\}_{i = 1}^N$  and 3D  $\{p_i\}_{i = 1}^N$  features, respectively.  $t_l$  is the CLIP's text embedding of the  $l$ -th class name. Note that CLIP and 2D network inputs are the
same images with different data augmentation. Thus we can utilize all CLIP's pseudo labels to train the 2D network.
In the second stage, we additionally include the self-training and cross-training process to reduce the error propagation flow of noisy prediction. Specifically, we first obtain the self-predicted pseudo-labels of 2D network  $\hat{P}_{x_i}^*$  and 3D network  $\hat{P}_{p_i}^*$ , respectively. Note that  $\hat{P}_{x_i} = \arg \max_l\psi (\theta_s(x_i),t_l)$ ,  $\hat{P}_{p_i} = \arg \max_l\psi (\varphi_s(p_i),t_l)$  and then they are refined to be  $\hat{P}_{x_i}^*$  and  $\hat{P}_{p_i}^*$  by SAM (Formula 4). Next, we randomly feed the 2D, 3D, and refined CLIP pseudo-labels to train the 2D and 3D networks. The objective function is as follows:
$$
\mathcal {L} _ {s 1} = \sum_ {i} C E \left(\theta_ {s} \left(x _ {i}\right), t _ {R _ {i} ^ {*}}\right) + \sum_ {i} C E \left(\varphi_ {s} \left(p _ {i}\right), t _ {R _ {i} ^ {*}}\right), \tag {6}
$$
where the pseudo-label  $R_{i}^{*}\in \{P_{x_{i}}^{*},P_{p_{i}}^{*},\hat{P}_{x_{i}}^{*},\hat{P}_{p_{i}}^{*}\}$  is randomly assigned to 2D and 3D networks with equal possibility during training.
Note that two training stages are seamless, i.e., we train 2D and 3D networks in the first stage with a few epochs to warm up the network and then seamlessly switch to the second training stage. In this way, we reduce the error propagation flow of noisy prediction and prevent the networks from overfitting the noisy labels.
Latent Space Consistency Regularization Inspired by the remarkable performance of zero-shot segmentation achieved by SAM, where objects are usually distinguished with clear boundaries, we consider how SAM endows the 2D and 3D networks with the robust segment anything capability to further resist the prediction noise. The intuitive solution is to transfer the knowledge from the SAM feature space to the image and point cloud feature spaces learned by target 2D and 3D networks.
Previous methods [17, 14, 13] employ InfoNCE loss for cross-modal knowledge transfer. These methods initially establish positive pixel-point pairs and negative pairs. They subsequently employ the InfoNCE loss to encourage the positive pairs to be closer and push away the negative pairs in the embedding space. However, Because precise object semantics are missing, these methods face a common optimization-conflict challenge. For instance, if two pixels have identical semantics but are wrongly assigned to different clusters, the InfoNCE loss tends to separate them, which is unreasonable and adversely affects downstream task performance [17, 14]. To this end, we choose to directly pull in the point feature with its corresponding pixel in the SAM feature space. The objective function is:
$$
\mathcal {L} _ {f} = \sum_ {i = 1} ^ {N} \left(1 - C O S \left(\theta_ {f} \left(x _ {i}\right), \frac {\phi_ {f} \left(s _ {i}\right)}{\| \phi_ {f} \left(s _ {i}\right) \| _ {2}}\right)\right) + \left(1 - C O S \left(\varphi_ {f} \left(p _ {i}\right), \frac {\phi_ {f} \left(s _ {i}\right)}{\| \phi_ {f} \left(s _ {i}\right) \| _ {2}}\right)\right), \tag {7}
$$
where  $\theta_{f}(\cdot),\varphi_{f}(\cdot)$  , and  $\phi_f(\cdot)$  indicate the linear mapping layers of 2D  $\{x_i\}_{i = 1}^N$  , 3D  $\{p_i\}_{i = 1}^N$  and SAM feature  $\{s_i\}_{i = 1}^N$  , with the same output feature dimensions  $K_{f}$  .  $COS(\cdot)$  is the cosine similarity. Note that the SAM feature is frozen during training.
Essentially, SAM has been trained on millions of images using large-scale masks, resulting in a highly robust feature representation contributing to its exceptional performance in zero-shot image segmentation. This robust feature representation can be used as a foundation for aligning the 2D and 3D features so that 2D and 3D paired features can be constrained into a single, unified SAM feature space to alleviate the problem of noisy predictions.