Title: Towards Label-free Scene Understanding by Vision Foundation Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: label-free

scene understanding
Abstract: Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot performance on image classification and segmentation tasks. However, the incorporation of CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, we investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CLIP and further exacerbated during the propagation from the 2D to the 3D domain. To tackle these challenges, we propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-free semantic segmentation with 28.4\% and 33.5\% mIoU on ScanNet, improving 4.7\% and 7.9\%, respectively. For nuImages and nuScenes datasets, the performance is 22.1\% and 26.8\% with improvements of 3.5\% and 6.0\%, respectively. Code is available. (https://github.com/runnanchen/Label-Free-Scene-Understanding)

Towards Label-free Scene Understanding by Vision Foundation Models

Abstract

Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot performance on image classification and segmentation tasks. However, the incorporation of CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, we investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CLIP and further exacerbated during the propagation from the 2D to the 3D domain. To tackle these challenges, we propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-free semantic segmentation with 28.4% and 33.5% mIoU on ScanNet, improving 4.7% and 7.9%, respectively. For nuImages and nuScenes datasets, the performance is 22.1% and 26.8% with improvements of 3.5% and 6.0%, respectively. Code is available. 2

1 Introduction

Scene understanding aims to recognize the semantic information of objects within their contextual environment, which is a fundamental task for autonomous driving, robot navigation, digital city, etc. Existing methods have achieved remarkable advancements in 2D and 3D scene understanding [110]. However, they heavily rely on extensive annotation efforts and often struggle to identify novel object categories that were not present in the training data. These limitations hinder their practical applicability in real-world scenarios where acquiring high-quality labelled data can be expensive and novel objects may appear [11-17]. Consequently, label-free scene understanding, which aims to perform semantic segmentation in real-world environments without requiring labelled data, emerges as a highly valuable yet relatively unexplored research topic.

Vision foundation models, e . g ., Contrastive Vision-Language Pre-training (CLIP) [18] and Segment Anything (SAM) [19], have garnered significant attention due to their remarkable performance in addressing open-world vision tasks. CLIP, trained on a large-scale collection of freely available imagetext pairs from websites, exhibits promising capabilities in open-vocabulary image classification. On

CLIP

SAM

Noisy pseudo-labels

Clean object masks

Images

Label-free 3D scene understanding

Label-free 2D scene understanding

Transfer

to 3D

3D ground truth

2D ground truth

the other hand, SAM learns from an extensive dataset comprising 1 billion masks across 11 million images, achieving impressive zero-shot image segmentation performance. However, initially designed for image classification, CLIP falls short in segmentation performance. Conversely, SAM excels in zero-shot image segmentation but lacks object semantics (Fig. 1). Additionally, both models are trained exclusively on 2D images without exposure to any 3D modal data. Given these considerations, a natural question arises: Can the combination of CLIP and SAM imbue both 2D and 3D networks with the ability to achieve label-free scene understanding in real-world open environments?

Despite recent efforts [20] leverage CLIP for image-based semantic segmentation, the pseudo labels generated by CLIP for individual pixels often exhibit significant noise, resulting in unsatisfactory performance. The up-to-date work [17] has extended this method to encompass 3D label-free scene understanding by transferring 2D knowledge to 3D space via projection. However, the pseudo labels assigned to 3D points are considerably noisier due to calibration errors, significantly limiting the accuracy of the networks. The primary challenge of utilizing vision foundation models for scene understanding is effectively supervising the networks using exceptionally noisy pseudo labels.

Inspired by SAM's impressive zero-shot segmentation capabilities, we propose a novel Cross-modality Noisy Supervision (CNS) framework incorporating CLIP and SAM to train 2D and 3D networks simultaneously. Specifically, we employ CLIP to densely pseudo-label 2D image pixels and transfer these labels to 3D points using the calibration matrix. Since pseudo-labels are extremely noisy, leading to unsatisfactory performance, we refine pseudo-labels using SAM's masks, producing more reliable pseudo-labels for supervision. To further mitigate error propagation and prevent the networks from overfitting the noisy labels, we consistently regularize the network predictions, i . e ., co-training the 2D and 3D networks using the randomly switched pseudo labels, where the labels are from the prediction of 2D, 3D, and CLIP networks. Moreover, considering that individual objects are well distinguished in SAM feature space, we use the robust SAM feature to consistently regularize the latent space of 2D and 3D networks, which aids networks in producing semantic predictions with more precise boundaries and further reduces label noise. Note that the SAM feature space is frozen during training, thus severed as the anchor metric space for aligning the 2D and 3D features.

To verify the label-free scene understanding capability of our method in 2D and 3D real open worlds, we conduct experiments on indoor and outdoor datasets, i . e ., ScanNet and nuScenes, where 2D and 3D data are simultaneously collected. Extensive results show that our method significantly outperforms state-of-the-art methods in understanding 2D and 3D scenes without training on any labelled data. Our 2D and 3D network achieves label-free semantic segmentation with 28.4% and 33.5% mIoU on ScanNet, improving 4.7% and 7.9%, respectively. For nuImages and nuScenes datasets, the performance is 22.1% and 26.8% with improvements of 3.5% and 6.0%, respectively. Quantitative and qualitative ablation studies also demonstrate the effectiveness of each module.

The key contributions of our work are summarized as follows.

In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data.

We propose a novel Cross-modality Noisy Supervision framework to effectively and synchronously train 2D and 3D networks with severe label noise, including prediction consistency and latent space consistency regularization schemes.

Experiments conducted on indoor and outdoor datasets show that our method significantly outperforms state-of-the-art methods on 2D and 3D semantic segmentation tasks.

2 Related Work

Scene Understanding. Scene understanding is a fundamental task for computer vision and plays a critical role for robotics, autonomous driving, smart city, etc. Significant advancements have been made by current supervised methods for 2D and 3D scene understanding [1-10, 21-34]. However, these methods heavily depend on extensive annotation efforts, which pose significant challenges when encountering novel object categories that were not included in the training data. In order to overcome these limitations, researchers have proposed self-supervised and semi-supervised methods [13, 14, 3554] to train networks in a more data-efficient manner. Nevertheless, these approaches struggle to handle novel objects that are not present in training data and they often exhibit subpar performance when faced with significant domain gaps. Alternatively, some methods [11, 12, 55-64] focus on open-world scene understanding that identifies novel object categories absents in the training data. However, these methods still require expensive annotation efforts on the close-set category data. Other methods [56, 17, 65-67] try to alleviate 3D annotation efforts by distilling knowledge from 2D networks. However, they either need 2D labelled data to train networks or struggle to achieve satisfactory results in the context of label-free 3D scene understanding.

Recently, vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) [18] and Segment Anything (SAM) [19] have garnered significant attention due to their remarkable performance in open-world vision tasks. MaskCLIP [20] extends the capabilities of CLIP to image semantic segmentation, achieving promising results in 2D scene understanding without labelled data. Furthermore, CLIP2Scene [17] expands upon MaskCLIP to enable 3D scene understanding through the use of a 2D-3D calibration matrix. In this study, our focus lies in label-free understanding using vision foundation models. Our ultimate goal is to perform semantic segmentation in real-world environments without relying on labelled data. We hope that our work will inspire and motivate further research in this area.

Label-noise Representation Learning. Label-Noise Representation Learning (LNRL) aims to train neural networks robustly in the presence of noisy labels. Numerous methods have been developed to tackle this challenge. Some methods [68-74] focus on designing accurate estimators of the noise transition matrix, which establish a connection between the clean class posterior and the noisy class posterior. Others [75-81] devise noise-tolerant objective functions through explicit or implicit regularization, enabling the learning of robust classifiers on noisy data. Furthermore, Some approaches [82-86] delve into the dynamic optimization process of LNRL based on the memorization effects, i . e ., deep models tend to initially fit easy (clean) patterns and gradually overfit complex (noisy) patterns. However, the above methods are insufficient when clean labels are unavailable, which makes them inadequate for addressing our problem with entirely noisy labels. Furthermore, these methods are primarily designed for a single modality, and the exploration of multiple modality scenarios is still an unexplored topic. In this paper, we study the LNRL across different modalities without any clean labels.

3 Methodology

The vision foundation models have demonstrated remarkable performance in open-world scenarios. However, while CLIP excels in zero-shot image classification, it tends to produce noisy masks in segmentation tasks. Conversely, SAM generates impressive object masks in zero-shot segmentation but lacks object semantics. Therefore, our research aims to mitigate the impact of noisy labels by leveraging the strengths of various modalities, such as images, point clouds, CLIP, and SAM. In what follows, we revisit the vision foundation models applied in open-vocabulary classification and semantic segmentation, then present the Cross-modality Noisy Supervision (CNS) framework in detail.

Consistency Regularization

Prediction

Consistency Regularization

Pseudo-labeling

and Refinement

2D-3D Projection

Projection

Augmentation

2D

3D

SAM

2D Image input

2D Image input

3D Point Cloud

2D Features

SAM Features

3D Features

2D Predictions

3D Predictions

Refined 2D pseudo-labels

Refined 3D pseudo-labels

CLIP's text embeddings

CLIP's text embeddings

Forward operation

Loss supervision

Label random switching

Knowledge distilling

CLIP 3D Predictions

SAM

Refinement

2D/3D Pseudo-labels

SAM

Refinement

CLIP 2D Predictions

CLIP

2D Image input

Transfer to 3D

Transfer to 3D

SAM

Refinement

SAM

Refinement

3.1 Revisiting Vision Foundation Models

Vision foundation model refers to powerful models pre-trained on large-scale multi-modal data and can be applied to a wide range of downstream tasks [87]. CLIP and SAM are well recognized due to their impressive open-world image classification and segmentation performance.

CLIP combines an image encoder (ResNet [88] or ViT [89]) and a text encoder (Transformer [90]), which independently map image and text representations to a shared embedding space. CLIP trains both encoders using a contrastive loss on a dataset of 400 million image-text pairs sourced from the Internet that contain diverse image classes and textual concepts. This training approach enables CLIP to achieve impressive performance in open-vocabulary recognition. CLIP utilizes a two-step process for zero-shot image classification. Initially, it obtains image embeddings and generates text embeddings by incorporating the class name into a predefined template. Then, it compares the similarity of image and text embeddings to determine the class. SAM serves as the fundamental model for image segmentation. It undergoes training using over 1 billion masks on a dataset of 11 million licensed images, enabling it to adapt seamlessly to new image distributions and tasks without prior exposure. Extensive evaluations have demonstrated that SAM's zero-shot performance is remarkable, frequently rivaling or surpassing the results of previous fully supervised approaches. Note that SAM works impressively on zero-shot image segmentation but lacks object semantics.

MaskCLIP [20] extends the CLIP's capabilities to image semantic segmentation. Specifically, MaskCLIP modifies the attention pooling layer of CLIP's image encoder to enable pixel-level mask prediction instead of global image-level prediction. CLIP2Scene [17] further expands MaskCLIP into 3D scene understanding by transferring the pixel-level pseudo-labels from the 2D image to the 3D point cloud using a 2D-3D calibration matrix. However, their performance is unsatisfactory due to the significant noise in the pseudo labels. To this end, we leverage the strengths of vision foundation models to supervise networks under extremely noisy pseudo labels.

3.2 Cross-modality Noisy Supervision

As illustrated in Fig. 2, we introduce a novel Cross-modality Noisy Supervision (CNS) method that combines CLIP and SAM foundation models to concurrently train 2D and 3D networks for label-free scene understanding. Our approach involves pseudo-labelling the 2D images and 3D points using CLIP. To address the noise in pseudo-labelling, we employ SAM's masks to refine the pseudo-labels. To further mitigate error propagation and prevent overfitting to noisy labels, we

propose a prediction consistency regularization mechanism. This mechanism randomly switches refined pseudo-labels from different modalities to co-train the 2D and 3D networks. Additionally, we consistently regularize the network's latent space using SAM's robust feature, which aids in generating more accurate predictions. In the following sections, we provide a comprehensive overview of our method, along with detailed explanations and insights.

Pixel-point Projection. Since vision foundation models are pre-trained on 2D images, we transfer knowledge from 2D images to 3D point clouds via pixel-point correspondences. Note that the dataset already provides the camera poses, so we obtain the pixel-point correspondence { c i , s i , x i , p i } N i =1 via simple projection, where c i , s i and x i indicates the i -th paired image pixel feature by CLIP, SAM, and 2D network, respectively. p i is the i -th paired point feature by the 3D network F 3 D and N is the number of pairs.

Pseudo-labeling by CLIP. We follow the pioneer methods, MaskCLIP and CLIP2Scene, that adopt CLIP to dense pseudo-label the 2D image pixels and transfer these labels to 3D points. Specifically, given the calibrated pixel-point features { c i , s i , x i , p i } N i =1 , we pseudo-label the pixel c i as follows:

where c i is the i -th pixel feature by CLIP's image encoder. t l is the CLIP's text embedding of the l -th class name into a pre-defined template. ψ is the dot product operation.

Since pixel-point features { c i , s i , x i , p i } N i =1 are spatial aligned, we can transfer the pseudo-label { P c i } N i =1 to all image pixels and point coulds, i . e .,

where { P s i } N i =1 , { P x i } N i =1 and { P p i } N i =1 are the transferred pseudo-labels from CLIP to SAM, 2D and 3D networks, respectively.

Label Refinement by SAM. Since CLIP is pre-trained on 2D images and text for zero-shot classification, it brings extreme noise to pixel-wise prediction. Nevertheless, another vision foundation model, Segment Anything (SAM), can provide impressive object masks in open-world environments, even rivaling or surpassing the previous fully supervised approaches. To this end, we consider refining the CLIP's pseudo-labels by SAM. Firstly, we follow the official implementation of SAM to generate all masks from the images. The mask ID of pixels are presented as { M s i } N i =1 , where M s i ∈ { 1 , 2 , ..., T } and T indicates the number of masks. Once again, we could transfer the masks to all image pixels and point clouds based on the pixel-point correspondence { c i , s i , x i , p i } N i =1 , i . e .,

where { M c i } N i =1 , { M x i } N i =1 and { M p i } N i =1 are the transferred masks from SAM to CLIP , 2D and 3D predictions, respectively.

We then adjust the pseudo-labels to be identical if they are within the same mask. For simplicity, we take the max voting strategy, i . e .,

where ✶ {·} is the indicator function. P ∗ c i is the SAM refined pseudo-label from P c i . To this end, all pixels and point pseudo-labels within the same mask will be corrected by the semantic label most pixels belonging to, i . e ., { P ∗ c i } N i =1 , { P ∗ x i } N i =1 and { P ∗ p i } N i =1 .

Prediction Consistency Regularization. After obtaining the refined pseudo labels of all image pixels and points { P ∗ c i } N i =1 , { P ∗ x i } N i =1 and { P ∗ p i } N i =1 , we consider how to train both 2D and 3D networks. The training process consists of two stages. In the first stage, we simultaneously supervise the 2D and 3D networks by { P ∗ x i } N i =1 and { P ∗ p i } N i =1 with Cross-entropy loss:

where θ s and φ s are linear mapping layers of 2D { x i } N i =1 and 3D { p i } N i =1 features, respectively. t l is the CLIP's text embedding of the l -th class name. Note that CLIP and 2D network inputs are the

same images with different data augmentation. Thus we can utilize all CLIP's pseudo labels to train the 2D network.

In the second stage, we additionally include the self-training and cross-training process to reduce the error propagation flow of noisy prediction. Specifically, we first obtain the self-predicted pseudolabels of 2D network ˆ P ∗ x i and 3D network ˆ P ∗ p i , respectively. Note that ˆ P x i = arg max l ψ ( θ s ( x i ) , t l ) , ˆ P p i = arg max l ψ ( φ s ( p i ) , t l ) and then they are refined to be ˆ P ∗ x i and ˆ P ∗ p i by SAM (Formula 4). Next, we randomly feed the 2D, 3D, and refined CLIP pseudo-labels to train the 2D and 3D networks. The objective function is as follows:

where the pseudo-label R ∗ i ∈ { P ∗ x i , P ∗ p i , ˆ P ∗ x i , ˆ P ∗ p i } is randomly assigned to 2D and 3D networks with equal possibility during training.

Note that two training stages are seamless, i . e ., we train 2D and 3D networks in the first stage with a few epochs to warm up the network and then seamlessly switch to the second training stage. In this way, we reduce the error propagation flow of noisy prediction and prevent the networks from overfitting the noisy labels.

Latent Space Consistency Regularization Inspired by the remarkable performance of zero-shot segmentation achieved by SAM, where objects are usually distinguished with clear boundaries, we consider how SAM endows the 2D and 3D networks with the robust segment anything capability to further resist the prediction noise. The intuitive solution is to transfer the knowledge from the SAM feature space to the image and point cloud feature spaces learned by target 2D and 3D networks.

Previous methods [17, 14, 13] employ InfoNCE loss for cross-modal knowledge transfer. These methods initially establish positive pixel-point pairs and negative pairs. They subsequently employ the InfoNCE loss to encourage the positive pairs to be closer and push away the negative pairs in the embedding space. However, Because precise object semantics are missing, these methods face a common optimization-conflict challenge. For instance, if two pixels have identical semantics but are wrongly assigned to different clusters, the InfoNCE loss tends to separate them, which is unreasonable and adversely affects downstream task performance [17, 14]. To this end, we choose to directly pull in the point feature with its corresponding pixel in the SAM feature space. The objective function is:

where θ f ( · ) , φ f ( · ) , and ϕ f ( · ) indicate the linear mapping layers of 2D { x i } N i =1 , 3D { p i } N i =1 and SAM feature { s i } N i =1 , with the same output feature dimensions K f . COS ( · ) is the cosine similarity. Note that the SAM feature is frozen during training.

Essentially, SAM has been trained on millions of images using large-scale masks, resulting in a highly robust feature representation contributing to its exceptional performance in zero-shot image segmentation. This robust feature representation can be used as a foundation for aligning the 2D and 3D features so that 2D and 3D paired features can be constrained into a single, unified SAM feature space to alleviate the problem of noisy predictions.

4 Experiments

To evaluate the superior performance and generalization capability of our method for scene understanding, we conduct experiments on both indoor and outdoor public datasets, namely ScanNet [91], nuScenes [92] and nuImages [93], and compare with current state-of-the-art approaches [20, 56, 17] on both 2D and 3D semantic segmentation tasks. Critical operations and losses of our method are also verified effectiveness by extensive ablation studies. In the following, we introduce datasets, implementation, comparison results, and ablation studies in detail.

Datasets. ScanNet [91] consists of 1,603 indoor scans, collected by RGB-D camera, with 20 classes, where 1,201 scans are allocated for training, 312 scans for validation, and 100 scans for testing. Additionally, we utilize 25,000 key frame images to train the 2D network. The nuScenes [92] dataset,

collected in traffic scenarios by LiDAR and RGB camera, comprises 700 scenes for training, 150 for validation, and 150 for testing, focusing on LiDAR semantic segmentation with 16 classes. To be more specific, we leverage a total of 24,109 sweeps of LiDAR scans for training and 4,021 sweeps for validation. Each sweep is accompanied by six camera images, providing a comprehensive 360-degree view. Note that the dataset does not provide image labels. Thus the quantitative evaluation of the paired images is not presented. The nuImages [93] dataset provides 93,000 2D annotated images sourced from a significantly larger dataset. This includes 67,279 images for training, 16,445 for validation, and 9,752 for testing. The dataset consists of 11 shared classes with nuScenes.

Implementation Details. Weutilize MinkowskiNet34 [95] as the 3D backbone and DeeplabV3 [96] as the 2D backbone in our approach, where DeeplabV3 model is pre-trained on the ImageNet dataset. Based on the MaskCLIP framework, we modify the attention pooling layer of CLIP's image encoder to extract dense pixel-text correspondences. Our framework is developed using PyTorch and trained on two NVIDIA Tesla A100 GPUs. During training, both CLIP and SAM are kept frozen. For prediction consistency regularization, we transition to stage two after ten epochs of stage one. To enhance the robustness of our model, we apply various data augmentations, such as random rotation along the z-axis and random flip for point clouds, as well as random horizontal flip, random crop, and random resize for images. For the ScanNet dataset, the training process takes approximately 10 hours for 30 epochs, with the image number set to 16. In the case of the nuScenes dataset, the training time is 40 hours for 20 epochs, with the image number set to 6.

CLIP

SAM

Ours w/o CNS

Ground truth

Input

Ours

4.1 Comparison Results

Our method was compared with several state-of-the-art methods, namely MaskCLIP, MaskCLIP+, OpenScene, and CLIP2Scene, on the ScanNet and nuScenes datasets. Note that their codes are publicly available. For a fair comparison, we share all experiment settings with the above methods, including data augmentations, training epochs, learning rate, and network backbones. The results, as depicted in Table 1, demonstrate that our method surpasses the performance of other approaches by a significant margin. In terms of label-free semantic segmentation, our 2D and 3D networks achieve remarkable results, achieving 28.4% and 33.5% mIoU on the ScanNet dataset, corresponding to improvements of 4.7% and 7.9%, respectively. Furthermore, on the nuScenes dataset, our method achieves a mIoU of 26.8% for 3D semantic segmentation, exhibiting a 6% improvement. Note that the nuScenes dataset does not provide image labels. Thus the quantitative evaluation of 2D semantic segmentation is not presented.

4.2 Ablation Study

We conduct a series of ablation experiments to evaluate and demonstrate the effectiveness of different modules within our framework, as shown in Table. 2. In this section, we present the ablation configuration and provide an in-depth analysis of our experimental results.

Effect of Cross-modality Noisy Supervision. In 2D scene understanding, the Baseline is CLIP. For 3D scene understanding, we directly project the 2D prediction to the 3D point cloud. We also conducted w/o CNS that the Cross-modality Noisy Supervision was not applied, i . e ., build upon Baseline, the 2D label is refined by SAM, and projected to 3D points. Besides, we impose the baseline method self-training with the CLIP pseudo-labels, termed base+CLIP . Building upon base+CLIP , we also impose Latent Space Consistency Regularization, termed base+CLIP+ LSCR . Experiments

Ours w/o CNS

Ground truth

CLIP

Ours

Input

show that SAM refinement promotes pseudo-label quality. Moreover, our Cross-modality Noisy Supervision can effectively supervise networks under extremely noisy pseudo labels. We also evaluate CNS qualitatively using the ScanNet dataset, as illustrated in Fig. 3 and 4. Our comprehensive approach demonstrates remarkable performance in label-free scene understanding, encompassing both 2D and 3D scenarios. We observed that our results are comparable to human annotations in numerous instances. Interestingly, the 2D and 3D networks exhibit the remarkable ability to "segment anything," which can be attributed to the knowledge transferred from SAM.

Effect of Label Refinement. To verify how the label refinement affects the performance, we do not use SAM to refine the pseudo-labels during training ( w/o Refine ); we perform prediction refinement in the final result ( Post-Refine ). Notably, we observed a significant drop in accuracy when SAM refinement was omitted during training. This suggests that SAM is crucial in enhancing the model's performance. However, we also noticed that post-refinement did not contribute to the performance. This can be attributed to the fact that the trained networks already produced semantic-consistent regions similar to those obtained with SAM refinement.

Effect of Latent Space Consistency Regularization. We utilize two main features to evaluate the latent space consistency regularization: the CLIP feature and the CLIP+SAM feature. In addition, we have experimented without any Latent Space Regularization ( None ). Results demonstrate that our full method incorporating the SAM feature performs best. On the other hand, we have observed that the CLIP feature alone is less effective in our task, primarily because the feature space it provides is not specifically designed for segmentation purposes.

Effect of Prediction Consistency Regularization. We conducted five configurations to examine the effects of prediction consistency regularization: 1) w/o CT : In this experiment, the 2D and 3D networks were not allowed to cross-train each other in the second training stage, i . e ., the 2D network's labels were randomly switched between CLIP and 2D prediction. Meanwhile, the 3D network's labels were randomly switched between CLIP and 3D prediction; 2) w/o SCT : This experiment eliminates self- and cross-training in 2D and 3D networks, i . e ., 2D and 3D network labels are derived solely from CLIP; 3) w/o CLIP : This experiment excludes the usage of CLIP's labels during the second training stage, i . e ., both the 2D and 3D network's labels were randomly switched between 2D and 3D prediction; And 4) w 2D , 5) w 3D : we employ either 2D or 3D network prediction to

synchrony train 2D and 3D networks, respectively. Experiments show that SAM refinement promotes pseudo-label quality. Besides, The experimental results indicate that cross-training, self-training, and label-switching effectively improve performance. Importantly, we observed that the method fails when CLIP's supervision is absent during the second training stage. This failure can be attributed to the tendency of the 2D and 3D networks to 'co-corrupt" each other with extremely noisy labels.

Effect of Images Numbers. We also show the performance when restricting image numbers to 8, 16, and 32, respectively. However, we observe that the performances are comparable. Therefore, for efficient training, we set the image number to 16.

5 Conclusion

In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data. To address noisy pseudo-labels, we introduce a crossmodality noisy supervision method to simultaneously train 2D and 3D networks, which involves the label refinement strategy, prediction consistency regularization, and latent space consistency regularization. Our method achieves state-of-the-art performance in understanding open worlds, as demonstrated by extensive experiments on indoor and outdoor datasets.

References

Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9939-9948, 2020.

Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. In Advances in Neural Information Processing Systems , pages 33330-33342, 2022.

Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and LiangChieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12475-12485, 2020.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9939-9948, 2021.

Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li. Point-to-voxel knowledge distillation for lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8479-8488, 2022.

Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In IEEE/CVF International Conference on Computer Vision , pages 16024-16033, 2021.

Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. (af)2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE Conference on Computer Vision and Pattern Recognition , pages 12547-12556, 2021.

Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In IEEE/CVF International Conference on Computer Vision , pages 228-240, 2023.

Fangzhou Hong, Lingdong Kong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3d and 4d panoptic segmentation via dynamic shifting network. arXiv preprint arXiv:2203.07186 , 2022.

Runnan Chen, Xinge Zhu, Nenglun Chen, Wei Li, Yuexin Ma, Ruigang Yang, and Wenping Wang. Zeroshot point cloud segmentation by transferring geometric primitives. arXiv preprint arXiv:2210.09923 , 2022.

Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In International Conference on 3D Vision , pages 992-1002, 2021.

Yueh-Cheng Liu, Yu-Kai Huang, Hung-Yueh Chiang, Hung-Ting Su, Zhe-Yu Liu, Chin-Tang Chen, ChingYu Tseng, and Winston H Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint arXiv:2104.04687 , 2021.

Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9891-9901, 2022.

Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21705-21715, 2023.

Lingdong Kong, Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation , pages 9338-9345, 2023.

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7020-7030, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision , pages 696-712, 2022.

Yiteng Xu, Peishan Cong, Yichen Yao, Runnan Chen, Yuenan Hou, Xinge Zhu, Xuming He, Jingyi Yu, and Yuexin Ma. Human-centric scene understanding for 3d large-scale scenarios. In IEEE/CVF International Conference on Computer Vision , pages 20349-20359, 2023.

Youquan Liu, Runnan Chen, Xin Li, Lingdong Kong, Yuchen Yang, Zhaoyang Xia, Yeqi Bai, Xinge Zhu, Yuexin Ma, Yikang Li, et al. Uniseg: A unified multi-modal lidar segmentation network and the openpcseg codebase. In IEEE/CVF International Conference on Computer Vision , pages 21662-21673, 2023.

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34:17864-17875, 2021.

Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 7262-7272, 2021.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290-1299, 2022.

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In IEEE/CVF International Conference on Computer Vision , pages 12179-12188, 2021.

Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14408-14419, 2022.

Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005.10821 , 2020.

Runnan Chen. Studies on attention modeling for visual understanding. HKU Theses Online , 2023.

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11686-11695, 2022.

Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In IEEE/CVF International Conference on Computer Vision , pages 19994-2000, 2023.

Runnan Chen, Penghao Zhou, Wenzhe Wang, Nenglun Chen, Pai Peng, Xing Sun, and Wenping Wang. Pr-net: Preference reasoning for personalized video highlight detection. In IEEE/CVF International Conference on Computer Vision , pages 7980-7989, 2021.

MMDetection3D Contributors. Mmdetection3d: Openmmlab next-generation platform for general 3d object detection, 2020.

Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Benchmarking 3d perception robustness to common corruptions and sensor failure. In International Conference on Learning Representations Workshops , 2023.

Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. In Advances in Neural Information Processing Systems , 2023.

Runnan Chen, Xinge Zhu, Nenglun Chen, Dawei Wang, Wei Li, Yuexin Ma, Ruigang Yang, Tongliang Liu, and Wenping Wang. Model2scene: Learning 3d scene representation via contrastive language-cad models pre-training. arXiv preprint arXiv:2309.16956 , 2023.

Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Anyview self-supervised object segmentation on complex scenes. In International Conference on Learning Representations , 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems , volume 33, pages 21271-21284, 2020.

Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems , volume 27, 2014.

Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In IEEE/CVF International Conference on Computer Vision , pages 1422-1430, 2015.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , pages 1597-1607, 2020.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems , volume 33, pages 9912-9924, 2020.

Runnan Chen, Xinge Zhu, Nenglun Chen, Dawei Wang, Wei Li, Yuexin Ma, Ruigang Yang, and Wenping Wang. Towards 3d scene understanding by referring synthetic models. arXiv preprint arXiv:2203.10546 , 2022.

Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsupervised learning of intrinsic structural representation points. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9121-9130, 2020.

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2613-2622, 2021.

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4248-4257, 2022.

Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi-supervised semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4268-4277, 2022.

Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems , 34:22106-22118, 2021.

Xiangde Luo, Minhao Hu, Tao Song, Guotai Wang, and Shaoting Zhang. Semi-supervised medical image segmentation via cross teaching between cnn and transformer. In International Conference on Medical Imaging with Deep Learning , pages 820-833. PMLR, 2022.

Wei Shen, Zelin Peng, Xuehui Wang, Huayu Wang, Jiazhong Cen, Dongsheng Jiang, Lingxi Xie, Xiaokang Yang, and Q Tian. A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(8):9284-9305, 2023.

Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with crossconsistency training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12674-12684, 2020.

Runnan Chen, Yuexin Ma, Lingjie Liu, Nenglun Chen, Zhiming Cui, Guodong Wei, and Wenping Wang. Semi-supervised anatomical landmark detection via shape-regulated self-training. Neurocomputing , 471:335-345, 2022.

Yuhang Lu, Qi Jiang, Runnan Chen, Yuenan Hou, Xinge Zhu, and Yuexin Ma. See more and know more: Zero-shot point cloud segmentation via multi-modal visual data. In IEEE/CVF International Conference on Computer Vision , pages 21674-21684, 2023.

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211.15654 , 2022.

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language-driven open-vocabulary 3d scene understanding. arXiv preprint arXiv:2211.16312 , 2022.

Luigi Riz, Cristiano Saltori, Elisa Ricci, and Fabio Poiesi. Novel class discovery for 3d point cloud semantic segmentation. arXiv preprint arXiv:2303.11610 , 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. arXiv preprint arXiv:2112.14757 , 2021.

Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations , 2022.

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems , 32, 2019.

Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmentation. Advances in Neural Information Processing Systems , 33:10317-10327, 2020.

Ping Hu, Stan Sclaroff, and Kate Saenko. Uncertainty-aware learning for zero-shot semantic segmentation. Advances in Neural Information Processing Systems , 33:21713-21724, 2020.

Hui Zhang and Henghui Ding. Prototypical matching and open set rejection for zero-shot semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 6974-6983, 2021.

Xu Yan, Jiantao Gao, Chaoda Zheng, Chaoda Zheng, Ruimao Zhang, Shenghui Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. In European Conference on Computer Vision , pages 677-695, 2022.

Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9891-9901, 2022.

Xidong Peng, Runnan Chen, Feng Qiao, Lingdong Kong, Youquan Liu, Tai Wang, Xinge Zhu, and Yuexin Ma. Sam-guided unsupervised domain adaptation for 3d segmentation. arXiv preprint arXiv:2310.08820 , 2023.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1944-1952, 2017.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning , pages 1789-1799. PMLR, 2020.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(3):447-461, 2015.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems , 32, 2019.

Brendan Van Rooyen and Robert C Williamson. A theory of learning with corrupted labels. J. Mach. Learn. Res. , 18(1):8501-8550, 2017.

Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu. Estimating noise transition matrix with label correlations for noisy multi-label learning. In Advances in Neural Information Processing Systems , 2022.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instancedependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning , pages 25302-25312. PMLR, 2022.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems , 26, 2013.

Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations , 2020.

Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly supervised learning and label noise robustness. In International Conference on Machine Learning , pages 708-717. PMLR, 2016.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances in Neural Information Processing Systems , 28, 2015.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018.

Florian Jell. Theoretical foundations. In Patent Filing Strategies and Patent Management: An Empirical Study , pages 4-14. Springer, 2012.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems , 31, 2018.

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics , pages 4313-4324. PMLR, 2020.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning , pages 233-242. PMLR, 2017.

Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Msr: Making self-supervised learning robust to aggressive augmentations. arXiv preprint arXiv:2206.01999 , 2022.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision , pages 213-229, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30, 2017.

Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5828-5839, 2017.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11621-11631, 2020.

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11621-11631, 2020.

Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters , 7:3795-3802, 2022.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method that utilizes the SAM and CLIP (Contrastive Language-Image Pretraining) models for understanding the 2D and 3D world in a label-free manner. The authors suggest a two-step process to improve the results obtained from CLIP.\n\nIn the first step, the authors generate a noisy output from the CLIP model, which is primarily designed for semantic tasks. This output may contain imperfections or inaccuracies due to the inherent limitations of CLIP.To refine the output and obtain more visually appealing semantic results, the authors employ a SAM network in the second step. The SAM network is designed to enhance the quality of the semantic results generated by CLIP.\n\nFurthermore, the paper introduces a Cross-modality Noise Supervision module, which aims to optimize the training of both the 2D and 3D models simultaneously. \nThe authors demonstrate promising results on datasets such as ScanNet and nuScenes. These datasets are commonly used in the field of computer vision and provide challenging scenarios for understanding the 2D and 3D world. The encouraging results obtained from these datasets suggest the effectiveness of the proposed approach in understanding and representing the visual and semantic aspects of the real world.\n\nSTRENGTHS:\nStrength:\n\n1. Paper pushes the current boundary of training models with label free training data, using pre-existing models which have shown good performance on zero-shot tasks.\n2. Proposes to seamlessly refine labels predicted by CLIP model to achieve good results in Indoor Dataset.\n3. Paper for most of the part is well written and has a definite flow to it.\n4. Beats existing method such MaskCLIP, CLIP2scene models comprehensively to get state-of-arts results.\n\nWEAKNESSES:\n1. Missing Ablation: Prediction Consistency Regularization, the second stage where there is random switching of pseudo labels between 2D, 3D and CLIP based intermediate outputs, is not well explained. \n\n2. Missing Ablation: Ablation study which shows what is the impact of this random switching with and without. We also need to see what if we only employ one of the three pseudo labels and not all three, what impact does that have. Seeing the impact of each individual is necessary to comment about its efficacy.\n\n3. Effect of Latent Space Consistency Regularization: There is only marginal improvement seen using SAM features to guide the feature of intermediate 2D/3D encoder features. Do the authors have intermediate results(qualitative), which justifies how SAM features actually help the 2D and 3D features ?\n\nTypos:\n1. Line no 169: is the doc product operation. I presume this is supposed to be dot product.\n2. In Fig:2, How do we achieve the Refined CLIP Pseudo Label and how is this different from Refined 2D Pseudo labels. Please add further information for this.\n\nQUESTIONS:\n1. Could the authors provide intermediate feature space visualization which shows comparative results with and without latent space regularization ? Since from the ablation experiments provided there seems to be marginal improvement in mIoU numbers both in the ScanNet and nuScenes dataset.\n\nLIMITATIONS:\n1. The method proposed assumes SAM to be perfect in sense of providing meaning masks and does not train the SAM model. But fails to give examples or quote scenarios where this might fail, considering SAM outputs a mask which is not meaningful or helping to noisy CLIP based labels.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work employs 2D segmentation foundation models to perform semantic segmentation for both 2D and 3D indoor scenes. The clip model provides a general understanding of the semantic content in each image, while the SAM model generates precise segmentation masks. By integrating these two models, this study demonstrates improved quantitative results on two datasets.\n\nSTRENGTHS:\n1. The significant enhancement in segmentation results is achieved by leveraging two distinctively trained 2D segmentation models, SAM and Clip, surpassing the previous approach that solely utilized the Clip model.\n\n2. It is reasonable to anticipate improved segmentation outcomes by combining two powerful models that provide both semantic understanding and precise mask contours. The merging of diverse masks through pooling is straightforward and appears to be the most effective element contributing to the performance gain.\n\nWEAKNESSES:\n1. Writing: The authors are encouraged to provide further clarification on the \"calibration matrix,\" as it serves as the initial step in connecting 2D pixels and 3D points. How was this connection established without knowledge of the depth of each pixel? Additionally, could you please clarify the meaning of \"Image Number\" mentioned in line 305 and indicate the corresponding table or figure for reference? A similar question arises regarding the ablation study mentioned in line 290.\n\n2. Clarity:  In the abstract, the authors claim that the proposed method outperforms the state-of-the-art by a significant margin, but the method's name is not mentioned. When the term \"calibration matrix\" is first introduced in line 51, could you please provide a clear explanation? Does the proposed method predict both 2D and 3D semantic label maps? In Figure 2, which model requires training/finetuning and which one is frozen? How is the SAM prompted to predict the segmentation maps?\n\n3. As a straightforward baseline, the authors could utilize Clip to generate coarse semantic masks and merge these masks with SAM segmentation results. It is recommended to present the performance in Table 2 and the corresponding figure. The suggested experiments should be evaluated on the large-scale Scannet dataset. Furthermore, if the 2D foundation models already perform well, the motivation behind the \"Prediction Consistency Regularization\" is not clear. Why is there a need to fine-tune your own model?\n\n4. Although the demonstrated segmentation results are close to the ground truth, the mIoU scores for both 2D and 3D remain below 35.\n\n5. The improvement achieved through \"Latent Space Consistency Regularization\" appears to be minimal for both 2D and 3D segmentation, despite being listed as an important contribution.\n\n6. Please include citations for NeRF-based and mesh-based scene segmentation:\n\n[1] Atlas: End-to-end 3d scene reconstruction from posed images (supervised and unsupervised)\n[2] Decomposing nerf for editing via feature field distillation\n[3] Nerf-sos: Any-view self-supervised object segmentation on complex scenes \n[4] Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation\n\nQUESTIONS:\nAs the reviewer's question mainly lies in the clarity and writing, please refer to the \"Weakness\" section.\n\nLIMITATIONS:\nPlease refer to the \"Weakness\" section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using SAM to improve the quality of segmentation masks generated by CLIPs. The refined masks are then used as pseudo-labels to train networks that excel at segmentation tasks, following a similar training process to MaskCLIP+. Based on this idea, a two-stage self-training process is proposed to simultaneously train 2D and 3D networks with SAM-refined masks and self-generated masks iteratively. To mitigate mask noise, the authors further propose latent space consistency regularization to transfer knowledge from the SAM feature space to segmentation networks. The final trained segmentation networks demonstrate impressive performance on both 2D and 3D segmentation datasets.\n\nSTRENGTHS:\n(1) The overall method is both simple and effective. Although the authors use SAM trained with labeled data, which means it is not strictly label-free, I believe that it is acceptable given the performance improvement and contribution to the community.\n\n(2) The performance is impressively prefect.\n\n(3) The ablation study proves the gains of the main contributions of the paper.\n\n(4) The paper is well-written and easy to read and understand except for some minor issues.\n\nWEAKNESSES:\n(1) In this paper, the authors chose DeeplabV3 as the 2D backbone and MinkowskiNet34 as the 3D backbone. However, previous works such as MaskCLIP used DeeplabV2 as the 2D backbone and CLIP2Scene used MinkowskiNet14 as the 3D backbone. It is obviously that the backbones used in this paper have better performance, so it may not be entirely fair to compare them with previous works.\n\n(2) I'm a bit confused about the details of the Label Refinement by SAM. SAM often outputs very fine-grained masks and produces multi-level masks, while CLIP also often outputs masks with a lot of discrete noise points. How to handle these situations? Can the author provide a simple pseudo-code to describe this part of the processing?\n\n(3) I think the baseline selected by this paper is not appropriate. While the main contributions of this paper are the improvement of pseudo-labeling quality and more effective training strategies, the baseline should be a basic method that incorporates both of these points. For example, I think that for 2D scenes, MaskCLIP+ is a better baseline than CLIP. \n\n(4) I think the presentation of Table 2 could be improved. Except for the experimental settings in the first row, the configurations in the other rows are abridged versions of the full configuration in a specific structure. To better present the impact of each structure on the final configuration, I suggest comparing the results of each ablation experiment with the performance drop of the full configuration results.\n\n(5) MaskCLIP++ -> MaskCLIP+.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method for jointly learning 2D and 3D segmentation models that can be queried with open-world text queries. The paper's main contribution is a new Cross-modality Noisy Supervision to reduce the influence of noise on the predictions. The proposed method uses a SAM model to predict masks for every pixel and then uses those noises to regularize CLIP's noisy dense pixel features. The results shown in the paper show that the proposed method outperforms baselines in both 2D and 3D segmentation. I have a couple of questions and suggestions to improve the paper and I am willing to increase my score if these are sufficiently addressed in the rebutal.\n\nSTRENGTHS:\nThe paper is overall well-written and mostly easy to follow (apart from the mathematical notation, see comments below). The paper provides a pretty comprehensive overview of the literature up until CVPR 2023, and I also liked the qualitative results and Figure 1, which was informative.\n\nWEAKNESSES:\nI have some suggestions (in no particular order) to improve the paper:\n\n1. I take some issue with the claim that the method is the first to combine SAM and CLIP for label-free scene understanding. Multiple different methods have combined 2D segmentation models with CLIP, and as these systems are built modularity, they have already been able to switch out their segmenters for SAM. Two papers that come to mind for this are OvSeg (2D segmentation) and ClipFusion (3D segmentation). I suggest that this claim be softened.\n\n2. Figure 2 is very non-intuitive in my opinion and I did not fully understand the flow of information even after multiple reads. Maybe it would be good to improve this figure.\n\n3. Section 3.2 seems to be overly mathy and the notation is extremely convoluted. Consider cleaning up the super/sub scripts, and I also have a few additional comments:\n* Equations 2 and 3 take a lot of space do not add anything to my understanding of the method. I suggest removing them. \n* It is not necessary to define the range of each variable. \n* Equations 5/6 are extremely convoluted. Consider simpler notation.\n* In equation 1, I think there is an error, and the dot product should be between c_i^T, t_l\n\n4. Regarding the results in Table 1, the OpenScene results seem to be much lower compared to the ones reported in the original paper. How can this be explained? Was the baseline retrained for this paper?\n\nQUESTIONS:\nI have a few clarification questions as well:\n\n1. Line 159: What is meant by pixel-point calibration? Are the camera poses not known? Is there an additional step required apart from simple projection? If not, this section can be shortened in my opinion.\n\n2. Line 160: What is meant by the domain gap here? And why does this matter in the first place? As I see it, the method tries to align the features from the 3D network to the 2D CLIP/SAM space, and the domain gap should not matter here, right?\n\n3. Line 209: In the current method, you use cosine similarity to force 2D/3D network outputs to be similar to the SAM feature output. Is this objective not in contrast to enforcing similarity between CLIP features and the 3D/3D network outputs? Especially when looking at the results in Table 2, this specific regularization does not seem to add a lot in terms of performance (Latent Space Consistency Regularization)\n\nLIMITATIONS:\nThe paper does not discuss limitations and broader impact in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}