Towards Label-free Scene Understanding by Vision Foundation Models
Authors: Runnan Chen
Youquan Liu
Lingdong Kong
Nenglun Chen
Xinge ZHU
Yuexin Ma
Tongliang Liu
Wenping Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: label-free
scene understanding
TL;DR: In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data.

Abstract: Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot performance on image classification and segmentation tasks. However, the incorporation of CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, we investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CLIP and further exacerbated during the propagation from the 2D to the 3D domain. To tackle these challenges, we propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-free semantic segmentation with 28.4\% and 33.5\% mIoU on ScanNet, improving 4.7\% and 7.9\%, respectively. For nuImages and nuScenes datasets, the performance is 22.1\% and 26.8\% with improvements of 3.5\% and 6.0\%, respectively. Code is available. (https://github.com/runnanchen/Label-Free-Scene-Understanding)

Towards Label-free Scene Understanding by Vision Foundation Models
Runnan Chen'
Youquan Liu
Lingdong Kong" Nenglun Chen' Wenping Wang"*
Xinge Zhu"
Yuexin M
Tongliang Liu
The University of Hong Kong
Hochschule Bremerhaven
3 National University of Singapore
"The Chinese University Hong Kong
"ShanghaiTech University Thhe University of Sydney
TTexas A&M University
Abstract
Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot perfor- mance image classification and segmentation tasks. However, the incorporation CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CL.IP and further exacerbated during the propagation from the 2D the 3D domain. To tackle these challenges, we propose novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths CLIP and SAM to supervise 2D and 3D networks simultancously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-fres freeeffree semantic segmentation with 28.4% and 33.5% mloU on ScanNet, improving 4.7% and 7.9%, respectively. For nulmages and nuScenes datasets, the performance is 22. 1 % and 26.8% with improvements of 3.5% and 6.0%, respectively. Code is available.?
Introduction
Scene understanding aims recognize the semantic information of objects within their contextual environment, which is fundamental task for autonomous driving, robot navigation, digital city, etc. Existing methods have achieved remarkable advancements 2D and 3D scene understanding [I- 10]. However, they heavily rely extensive annotation efforts and often struggle to identify novel object categories that were not present the training data. These limitations hinder their practical applicability in real-world scenarios where acquiring high-quality labelled data can be expensive and novel objects may appear [1 1-1]. Consequently, label-free scene understanding, which aims to perform semantic segmentation in real-world environments without requiring labelled data, emerges as a highly valuable yet relatively unexplored research topic.
Vision foundation models, e.g., Contrastive Vision-Language Pre-training (CLIP) [18] and Segment Anything (SAM) [19], have garnered significant attention due to their remarkable performance in addressing open- open-world vision tasks. CLIP, trained on a large-scale collection of freely available image- text pairs from websites, exhibits promising capabilities ooen-vocay image classification. On
"Corresponding authors.
phttps://gitaie com/ wwwnanchennnaaaeeee..e Free-FFreeeeeeee Scene Serree-Seeee-uddeesseeeeeee Understanding.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Noisy pooado-lahels
Labcl-fre 3D spone unlersianding
20 ground truth
Trruirr M.
NAA
Intages
Label-fre: sccne understanding
ground trath
Figure I: We study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data. To accomplish this, we introduce a novel framework called Cross-modality Noisy Supervision (CNS). By effectively harnessing the strengths of CLIP and SAM, our approach simultancously trains 2D and 3D networks, yielding remarkable performance.
other hand, SAM learns from an extensive dataset comprising billion masks across million images, achieving impressive zero-shot image segmentation performance. However, initially designed for image classification, CLIP falls short in segmentation performance. Conversely, SAM excels in zero-shot image segmentation but lacks object semanties (Fig. 1). Additionally, both models trained exclusively on 2D images without exposure to any 3D modal data. Given these considerations, natural question arises: Can the combination of CLIP and SAM imbue both 2D and 3D networks with the ability to achieve label-free scene understanding in real-world open environments?
Despite recent efforts [20] leverage CLIP for image-based semantic segmentation, the pseudo labels generated by CLIP for individual pixels often exhibit significant noise, resulting in unsatisfactory performance. The up-to-date work [17] has extended this method to encompass 3D label-free scene understanding by transferring 2D knowledge assigned to 3D points are considerably noisier due calibration errors, significantly limiting the accuracy of the networks. The primary challenge of utilizing vision foundation models for scene understanding is effectively supervising the networks using exceptionally noisy pseudo labels.
space via projection. However, the pseudo labels
Inspired by SAM's impressive zero-shot segmentation capabilities, we propose novel Cross-modality Noisy Supervision (CNS) framework incorporating CLIP and SAM to train 2D and 3D networks simultaneously. Specifically, we employ CLIP to densely pseudo-labe 2D image pixels and transfer these labels to 3D points using the calibration matrix. Since pseudo-labels are extremely noisy, leading to unsatisfactory performance, we refine pseudo-labels using SAM's 's masks, producing more reliable pseudo-labels for supervision. To further mitigate error propagation and prevent the networks from overfitting the noisy labels, we consistently regularize the network predictions, i.e., co-training the 2D and 3D networks using the randomly switched pscudo labels, where the labels are from the prediction of 2D,31 3D, and CLIP networks. Moreover, considering that individual objects are well distinguished in SAM feature space, we use the robust SAM feature to consistently regularize the latent space of 2D and 3D networks, which aids networks in producing semantic predictions with more precise boundaries and further reduces label noise. Note that the SAM feature space is frozen during training, thus severed as the anchor metric space for aligning the 2D and 3D features.
To verify the label-free scene understanding capability of our method in 2D and 3D real open worlds, we conduct experiments on indoor and outdoor datasets, i.e., ScanNet and nuScenes, where 2D and 3D data are simultaneously collected. Extensive results show that our method significantly outperforms state-of-the-an methods in understanding 2D and 3D scenes without training on any labelled data. Our 2D and 3D network achieves label-free semantic segmentation with 28.4% and 33.5% mloU on ScanNet, improving 4.7% and 7.9%, respectively. For nulmages and nuScenes datasets, the performance is 22. 1% and 26.8% with improvements of 3.5% and 6.0%, respectively. Quantitative and qualitative ablation studies also demonstrate the effectiveness of each module.
The key contributions of our work are summarized as follows.
In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying labelled data.

---

We propose a novel Cross-modalit, Noisy Supervision framework to effectively and syn- chronously train 2D and 3D networks with severe label noise, including prediction consis- tency and latent space consistency regularization schemes.
Experiments conducted on indoor and outdoor datasets show that our method significantly outperforms state-of-the-art methods on 2D and 3D semantic segmentation tasks.
2
Related Work
Scene Understanding. Scene understanding is a fundamental task for computer vision and plays a critical role for robotics, autonomous driving, smart city, etc. Significant advancements have been made by current supervised methods for 2D and 3D scene understanding [1-10, 21-34]. However, these methods heavily depend extensive annotation efforts, which pose significant challenges when encountering novel object categories that were not included in the training data. In order to overcome these limitations, researchers have proposed self- supervisdd and remi-supervised methods [13, 14, 35- 54] to train networks in a more data-efficient manner. Nevertheless, these approaches struggle to handle novel objects that when faced with significant domain gaps. A Alternatively, some methods [11, 12, 55-64] focus on open-world scene understanding that identifies novel object categories absents in the training data. However, these methods still require expensive annotation efforts on the close-set category data. Other methods [56, 17, 65-67] try alleviate 3D annotation efforts by distilling knowledge from 2D networks. However, they either need 2D labelled data to train networks or struggle to achieve satisfactory results in the context of label-free 3D scene understanding.
present in training data and they often exhibit subpar performance
Recently, vision foundation models such as Contrastive Vision- Laiiuuge Pre-training (CLIP) [18] and Segment Anything (SAM) [19] have garnrred significant attention due to their remarkable performance in open-world vision tasks. MaskCLIP [20] extends the capabilities of CLIP to image semantic segmentation, achieving promising results in 2D scene understanding without labelled data. Furthermore, CLIP2Scene [17] expands upon MaskCLIP to enable 3D scene understanding through the use of 2D-3D calibration matrix. In this study, our focus lies in label-free understanding using vision foundation models. Our ultimate goal is to perform semantic segmentation in real-world environments without relying on labelled data. We hope that our work will inspire and motivate further research in this area.
Label-noise Representation Learning. Label-Noise Representation Learning (LNRL) aims train neural networks robustly in the presence of noisy labels. Numerous methods have been developed to tackle this challenge. Some methods [68-74] focus on designing accurate estimators of the noise transition matrix, which establish a connection between the clean class posterior and the noisy class posterior. Others [75-81] devise noise-toleran objective functions through explicit or implicit regularization, enabling the learning of robust classifiers on noisy data. Furthermore, Some approaches [82-86] delve into the dynamic optimization process of LNRL based on the memorization effects, i.e., deep models tend to initially fit easy (clean) patterns and gradually overfit complex (noisy) patterns. However, the above methods are insufficient when clean labels are unavailable, which makes them inadequate for addressing our problem with entirely noisy labels. Furthermore, these methods are primarily designed for a single modality, and the exploration of multiple modality scenarios is still an unexplored topic. In this paper, we study the LNRL across different modalities without any clean labels.
3 Methodology
The vision foundation models have demonstrated remarkable performance in open-world scenarios. However, while CLIP excels in zero-shot image classification, it tends to produce noisy masks in segmentation tasks. Conversely, SAM generates impressive object masks zero-shot segmentation but lacks object semantics. Therefore, our research aims to mitigate the impact of noisy labels by leveraging the strengths of various modalities, such as images, point clouds, CLIP, and SAM. In what follows, we revisit the vision foundation models applied in voen-vocybulary classification and semantic segmentation, then present the Cross-modality Noisy Supervision (CNS) framework in detail.
3

---

20-30 Projection
Latent Space Consistency Segnlarization
Cousistency Regulartsttion
Prediction
Preudn-Saheeling Refinement
Sald Refinoment
CLIP's not uniolling
Irage imput
ID Feomups
BD Profictiona
Refinal 30 pacstteliccl
Aogmusiiaiin
S.ld Eilinonnntt
2D Image Input
SAM Feaudees
SAM Madks
20:3D Paedhn
2D Imsage impat
Projection
Sald Relivomont
Trangfir
CLIP's lon onbeddiing
Sdl Astinonunt
Cloud
Refinal 30 pansit-liscl
Prodictions
Forward
sipursicone
Kesmslogge
Figure Illustration of the overall framework. Specifically, we employ CLIP to densely pseudo-label 2D image pixels and transfer these labels to 3D points using the calibration matrix. Acknowledging that network predictions and pseudo-labels are extremely noisy, we refine pseudo-labels using SAM's masks for more reliable supervision. To mitigate error propagation and prevent the networks from overfitting the noisy labels, we randomly switch the different refined pseudo-labels co-training the 2D and 3D networks. Besides, we consistently regularize the network's ' latent space using SAM's robust feature, which benefits the networks to produce clearer predictions.
3.1 Revisiting Vision Foundation Models
Vision foundation model refers powerful models pre-trained on scale- multi-modal data and can be applied to a wide range of downstream tasks [87]. CLIP and SAM are well recognized due to their impressive open-wor wpeldworld image classification and segmentation performance.
CLIP combines an image encoder (ResNet [88] or ViT [89]) and text encoder (Transformer [90]), which independently map image and text representations to shared embedding space. CLIP trains both encoders using a contrastive loss on dataset of 400 million image-text pairs sourced from the Internet that contain diverse image classes and textual concepts. This training approach enables CLIP to achieve impressive performance
open-vocabulary recognition. CLIP utilizes a two-step process for zero-shot image classification. Initially, it obtains image embeddings and generates text embeddings by incorporating the class name into a predefined template. Then, compares the similarity of image and text embeddings to determine the class. SAM serves the fundamental model for image segmentation. undergoes training using over 1 billion masks on a dataset of 11 million licensed images, enabling it to adapt seamlessly to new image distributions and tasks without prior exposure. Extensive evaluations have demonstrated that SAM's zero-shot performance remarkable, frequently rivaling surpassing the results of previous fully supervised approaches. Note that SAM works impressively on zero-shot image segmentation but lacks object semantics.
MaskCLIP [20] extends the CLIP's capabilities to image semantic segmentation. Specifically, MaskCLIP modifies the attention pooling layer of CLIP's image encoder to enable pixel-level mask prediction instead of global image-level prediction. CLIP2Scene [17] further expands MaskCLIP into 3D scene understanding by transferring the pixel-level pseudo-labek from the 2D image to the 3D point cloud using 2D-3D calibration matrix. However, their performance is unsatisfactory due to the sigmificant noise in the pseudo labels. To this end, we leverage the strengths of vision foundation models to supervise networks under extremely noisy pseudo labels.
3.2 Cross-modality Noisy Supervision
As illustrated in Fig. 2, we introduce a novel Cross- Noisy Supervision (CNS) method that combines CLIP and SAM foundation models to concurrently train 2D and networks for label-free scene understanding. Our approach involves pseudo-labelling the 2D images and 3D points using CLIP. To address the noise in pseudo-labelling, we employ SAM's masks refine the pscudo-labels. To further mitigate error propagation and prevent overfitting noisy labels,

---

propose prediction consistency regularization mechanism. This mechanism randomly switches refined pseudo-labels from different modalities co-train the 2D and 3D networks. Additionally, we consistently regularize the network's latent space using SAM's robust feature, which aids in generating more accurate predictions. In the following sections, we provide a comprehensive overview of our method, along with detailed explanations and insights.
Pixel-point Projection. Since vision foundation models are pre-trained on 2D images, we transfer knowledge from 2D images to 3D point clouds via pixel-point correspondences. Note that the dataset already provides the camera poses, we obtain the pixel-point correspondence {cs, 8/y Fe Ps N via simple projection, where C₁, 8; and z; indicates the i-th paired image pixel feature by CLIP, SAM, and 2D network, respectively. P: the i-th paired point feature by the 3D network F3D and N the number of pairs.
Pseudo-labeling by CLIP We follow the pioneer methods, MaskCLIP and CLIP2Scene, that adopt CLIP to dense pseudo-label the 2D image pixels and transfer these labels to 3D points. Specifically, given the calibrated pixel-point features {cz, St, Fe, Pi ) pseudo-label the pixel e; as follows:
Pa arg (c .t)).
(1)
where a is the i-th pixel feature by CLIP's image encoder. t is the CLIP'' text embedding of the I-th class name into a pre-defined template. &b is the dot product operation.
Since pixel-point features {cz, 8i, Ts, p: P {Po 1 to all image pixels and point coulds, i.e.,
are spatial aligned, we can transfer the pseudo-label
Ps. = Pz. = P
(2)
where { (. (Pr.) and { are the transferred pseudo-labels from CLIP to SAM, 2D
and 3D networks, respectively.
Label Refinement by SAM. Since CLIP is pre-trained on 2D images and text for zero-shot classification, it brings extreme noise to pixel-wise prediction. Nevertheless, another vision foundation model, Segment Anything (SAM), can provide impressive object masks open-world environments, even rivaling or surpassing the previous fully supervised approaches. To this end, we consider refining the CL.IP's pseudo-labels by SAM. Firstly, we follow the official implementation of SAM to generate all masks from the images. The mask ID of pixels presented as {M,, Ji=1, where Ma E {1, 2, ..., T} and T indicates the number of masks. Once again, we could transfer the masks to all image pixels and point clouds based on the pixel-point correspondence {c,, 8,, Ti, pt p i.e.
J.N.
Mc; = Ma, = Mp: = Ma,,
where (Ma, i, (Mr, Y and (Mp, AMM.le. 1 are the transferred masks from SAM to CLIP, 2D and 3D predictions, respectively.
We then adjust the laeuls-labels to be identical if they are within the same mask. For simplicity, we take the max voting strategy, iee.
P = arg max
Σ
I[Po =1}
My.Me Mc,
where (.) is the indicator function. Ps is the SAM refined pseudo-label from Pa.- To this end, all pixels and point pseudo-labels within the same mask will be corrected by the semantic label most pixels belonging to, i.e., {Po, J.N, {Pa, J.W. and {Pp)
Consistency Regularization. After obtaining the refined pseudo labels of all image pixels and points J. networks. The training process consists of two stages. In the first stage, we simultaneously supervise the 2D and 3D networks by (P*, Jiml and {P
Prediction
and {P= J. we consider how to train both 2D and 3D
1 with Cross-entropy loss:
exp(v(<, by), Σ, exp(((I,
L. = ECE(O((E)))))) E CE(
CE(,,))))))) -log
(5)
where 0g and y are linear mapping layers of 2D [ and 3D {p:) N features, respectively. I is the CLIP's text embedding of the (-th class name. Note that CLIP and 2D network inputs are the
5

---

same images with different data augmentation. Thus we can utilize all CLIP's pseudo labels to train the 2D network.
In the second stage, we additionally include the self-training and cross- trainnn process to reduce the error propagation flow of noisy prediction. Specifically, we first obtain the self- eeff-predicted pseudo- labels 2D network Pr, and 3D network Pp. respectively. Note that Po: arg max, (x.(,, 44), Pp. arg max, () (p:), ta) and then they are refined to be Pr. and p. by SAM (Formula 4). Next, we randomly feed the 2D, 3D, and refined CLIP pseudo-labels to train the 2D and 3D networks. The objective function is as follows:
L1 E CCE()EE))) (CE(()(A))))))
(6)
where the pseudo-label R. {Pa, Pp Pr, Pr, is randomly assigned to 2D and 3D networks with equal possibility during training.
Note that two training stages are seamless, ie.. we train 2D and 3D networks in the first stage with a few epochs to warm up the network and then seamlessly switch to the second training stage. In this way, we reduce the error propagation flow of noisy prediction and prevent the networks from overfitting the noisy labels.
Latent Space Consistency Regularization Inspired by the remarkable performance of zero-shot segmentation achieved by SAM, where objects are usually distinguished with clear boundaries, we consider how SAM endows the 2D and 3D networks with the robust segment anything capability to further resist the prediction noise. The intuitive solution is transfer the knowledge from the SAM feature space to the image and point cloud feature spaces learned by target 2D and 3D networks.
loss for cross moosl knowledge transfer. These methods initially establish positive pixel-point pairs and negative pairs. They subsequently employ the InfoNCE loss encourage the positive pairs to be closer and push away the negative pairs in the embedding space. However, Because precise object semantics are missing, these methods face a common optimization-conffict challenge. For instance, if two pixels have identical semantics but wrongly assigned to different clusters, the InfoNCE loss tends to separate them, which is unreasonable and adversely affects downstream task performance [17, 14]. To this end, we choose to directly pull in the point feature with its corresponding pixel in the SAM feature space. The objective function is:
Previous methods [17, 14, 13] employ InfoNCE
6 )) + (1 c,= T--EacosO0/L Toy(s:)]12
0y(s.)
(7) where (y) (-)(), and (-) indicate the linear mapping layers of 2D {x;] 3D (p.).v and SAM feature {ss 1N with the same output feature dimensions Ky. COS(.) :) is the cosine similarity. Note that the SAM feature is frozen during training.
(1- (p)))))))) [<))(=))] )),
Essentially, SAM has been trained on millions of images using large-scale masks. resulting in a highly robust feature representation contributing to its exceptional performance in zero-shot image segmentation. This robust feature representation can used as foundation for aligning the 2D and 3D features so that 2D and 3D paired features can be constrained into single, unified SAM feature space to alleviate the problem of noisy predictions.
4 Experiments
To evaluate the superior performance and generalization capability of our method for scene under- standing, we conduct experiments on both indoor and outdoor public datasets, namely ScanNet [91], nuScenes [92] and nulmages [93], and compare with current state-of-the-a art approaches [20, 56, 17] on both 2D and 3D semantic segmentation tasks. Critical operations and losses of our method are also verified effectiveness by extensive ablation studies. In the following, we introduce datasets, implementation, comparison results, and ablation studies in detail.
Datasets. ScanNet [91] consists of 1,603 indoor scans, collected by RGB-D camera, with 20 classes, where 1,201 scans are allocated for training, 312 scans for validation, and 100 scans for testing. Additionally, utilize 25, 25,000 key frame images to train the 2D network. The nuScenes [92] dataset,

---

Table 1: Comparison (mloU) with current state-of-the-arl label-free methods for semantic segmenta- tion tasks on the ScanNet [91], nulmages [93] and nuScenes [94] dataset.
ScanNet
nulmages
nuScenes
Methods
Publication
2D
3D
2D
3D
MaskCLIP [20]
ECCV 2022
17.3
14.2
14.1
12.8
MaskCLIP+ [20]
ECCV 2022
20.3
21.6
17.3
15.3
OpenScene [56]
CVPR 2023
14.2
16.8
12.4
14.6
CLIP2Scene [17]
CVPR 2023
23.7
25.6
18.6
20.8
Ours
28.4
33.5
22.1
26.8
Table 2: Ablation study on the ScanNet [94] val set for label-free 2D and 3D semantic segmentation.
Ablation Target
Setting
2D mloU (%)
3D mloU (%
Baseline
17.3
14.2
w/o CNS
24.5 (47.20)
19.4 (45.20)
Cross-modality Noisy Supervision
Base+CLIP
20.4 (43.10)
23.8 (49.60)
Base+CLIP+LSS
22.5
(45.20)
28.7 (+14.6)
w/o Refine
23.7 (+6.40)
30.6 (+16.4)
Label Refinement
Post-Refine
28.1 (+10.8)
33.2 (4119.0)
w/o CT
27.7 (+10.1)
31.0 (+16.88
w/o SCT
24.7 (47.10)
30.6 (+16.4)
Prediction Consistency Regularization
w/o CLIP
4.7 (-12.6)
5.1 (-9.1)
w 2D
6.9 (-10.4)
8.8 (-5.4)
w 3D
4.8 (-12.5)
6.4 (-7.8)
CLIP
27.3 (410.0)
32.4 (+18.2)
Latent Space Consistency Regularization
SAM+CLIP
27.8 (+10.5)
32.5 (418.3)
None
27.1
(+9.80)
32.0
(+17.8)
27.9 (+10.6)
33.0 (+18.8)
Image Numbers
16
28.4 (+111)
33.5 (+19.3)
32
28.3 (+11.0)
33.3 (419.1)
Full Configuration
Ours
28.4 (411.))
33.5
(419.3)
collected in traffic scenarios by LiDAR and RGB camera, comprises 700 scenes for training, 150 for validation, and 150 for testing, focusing LiDAR semantic segmentation with 16 classes. To be more specific, we leverage total of 24, 109 sweeps of LiDAR scans for training and 4,021 sweeps for validation. Each sweep accompanied by six camera images, providing a comprehensive 360-degree view. Note that the dataset does not provide image labels. Thus the quantitative evaluation of the paired images is not presented. The nulmages [93] dataset provides 93,000 2D annotated images sourced from a significantly larger dataset. This includes 67, 27,279 images for training, 16,445 for validation, and 9,752 for testing. The dataset consists of 11 shared classes with nuScenes.
Implementation Details. We utilize MinkowskkNer134 [95] as the 3D backbone and Deeplab [96] as the 2D backbone in our approach, where Veeplab model is me-trained on the ImageNet dataset. Based on the MaskCLIP framework, we modify the attention pooling layer of CLIP's image encoder extract dense pixel-text correspondences. Our framework is developed using PyTorch and trained on two NVIDIA Tesla A100 GPUs. During training, both CLIP and SAM are kept frozen. For prediction consistency regularization, we transition to stage two after ten epochs of stage one. To enhance the robustness of our model, we apply various data augmentations, such as random rotation along the z-axis and random flip for point clouds, as well as random horizontal flip, random crop, and random resize for images. For the ScanNet dataset, the training process takes approximately 10 hours for epochs, with the image number set to 16. In the case of the nuScenes dataset, the training time is 40 hours for 20 epochs, with the image number set to 6.

---

Input
SAM
CLIP
Ours w/o CNS
Ours
Ground truth
Qualitative results of label-free semantic segmentation on the ScanNet 25k images dataset. From the left to the right column are the input, prediction by SAM, CLIP, ours w/o CNS, our full method, and ground truth, respectively.
Figure 3:
4.1 Comparison Results
Our method compared with several state-of-the-ar methods, namely MaskCLIP, MaskCLIP+, OpenScene, and CLIP2Scene, on the ScanNet and nuScenes datasets. Note that their codes are publicly available. For fair comparison, we share all experiment settings with the above methods, including data augmentations, training epochs, learning rate, and network backbones. The results, as depicted in Table 1, demonstrate that our method surpasses the performance of other approaches by significant margin. In terms of label-free semantic segmentation. our 2D and 3D networks achieve remarkable results, achieving 28.4% and 33.5% mIoU on the ScanNet dataset, corresponding to improvements of 4.7% and 7.9%, respectively. Furthermore, on the nuScenes dataset, our method achieves a mloU of 26.8% for 3D semantic segmentation, exhibiting a 6% improvement. Note that the nuScenes dataset does not provide image labels. Thus the quantitative evaluation of 2D semantic segmentation not presented.
4.2 Ablation Study
We conduct series of ablation experiments to evaluate and demonstrate the effectiveness of different modules within our framework, as shown in Table. 2. In this section, we present the ablation configuration and provide an in-depth analysis of our experimental results.
Effect of Cross-modality Noisy Supervision. In 2D scene understanding, the Baseline CLIP. For 3D scene understanding, we directly project the 2D prediction to the 3D point cloud. We also conducted w/o CNS that the Cross-modality Noisy Supervision was not applied, i.e., build upon Baseline, the 2D label is refined by SAM, and projected to 3D points. Besides, we impose the baseline method self-training with the CLIIP pseudo-labels, termed base+CLIP. Building upon base+CLIIP also impose Latent Space Consistency Regularization, termed base+CLIP. LSCR. Experiments

---

Input CLIP
Ours w/o CNS
Ours
Ground truth
BEEEE
Figure Qualitative results of label-free point cloud-based 3D semantic segmentation on the ScanNet dataset. From the left to the right are the input, prediction by CLIP, ours w/o CNS, our full method, and ground truth, respectively.
show that SAM refinement promotes pseudo-label quality. Moreover, Cross- Cross-modality Noisy Supervision can effectively supervise networks under extremely noisy pseudo labels. We also evaluate CNS qualitatively using the ScanNet dataset, as illustrated in Fig. and Our comprehensive approach demonstrates remarkable performance in label-free scene understanding, encompassing both 2D and 3D scenarios. We observed that our results comparable to human annotations in numerous instances. Interestingly, the 2D and 3D networks exhibit the remarkable ability to "segment anything," " which can be attributed to the knowledge transferred from SAM.
Effect of Label Refinement. To verify how the label refinement affects the performance, we do not use SAM refine the lseudo-labels during training (w/o Refine); we perform prediction refinement in the final result (Post- Reftne.. Notably, we observed significant drop in accuracy when SAM refinement was omitted during training. This suggests that SAM is crucial enhancing the model's performance. However, we also noticed that post-refinement did not contribute to the performance. This can be attributed to the fact that the trained networks already produced semantic-consistent regions similar to those obtained with SAM refinement.
Effect of Latent Space Consistency Regularization. We utilize two main features to evaluate the latent space consistency regularization: the CLIP feature and the CLIP+SAM feature. In addition, we have experimented without any Latent Space Regularization (None). Results demonstrate that our full method incorporating the SAM feature performs best. On the other hand, we have observed that the CLIP feature alone is less effective in our task, primarily because the feature space it provides is specifically designed for segmentation purposes.
Effect of Prediction Consistency Regularization. We conducted five configurations to examine the effects of prediction consistency regularization: 1) w/o CT: In this experiment, the 2D and 3D networks were not allowed to cross-train each other in the second training stage, i.e., the 2D network's 's labels were randomly switched between CLIP and 2D prediction. Meanwhile, the 3D network's labels were randomly switched between CLIP and 3D prediction; 2) w/o SCT: This experiment eliminates self- and cross-training in 2D and 3D networks, i.e., and 3D network labels are derived solely from CLIP: 3) w/o CLIP: This experiment excludes the usage of CLIP's labels during the second training stage, i.e., both the 2D and 3D network's 's labels were randomly switched between 2D and 3D prediction; And 4) 2D, 5) w 3D: employ either 2D or 3D network prediction to

---

synchrony train 2D and 3D networks, respectively. Experiments show that SAM refinement promotes pseudo-label quality. Besides, The experimental results indicate that tross-training, self- self-training, and label-switching effectively improve performance. Importantly, we observed that the method fails when CLIP's supervision absent during the second training stage. This failure can attributed to the tendency of the 2D and 3D networks to "co-corrupt" each other with extremely noisy labels.
Effect of Images Numbers. We also show the performance when restricting image numbers to 16, and 32, respectively. However, we observe that the performances are comparable. Therefore, for efficient training, we set the image number 16.
Conclusion
In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data. To address noisy pseudo-labels, we introduce a cross- modality noisy supervision method to simultaneously train 21> and 3D networks, which involves the label refinement strategy, prediction consistency regularization, and latent space consistency regularization. Our method achieves state-of-the-art performance in understanding open worlds, as demonstrated by extensive experiments on indoor and outdoor datasets.
Acknowledgements
Yuexin Ma was partially supported by NSFC (No.62206173), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Frontiers Science Center of Human-centers Artificial Intelligence (ShangHAI). Tongliaann Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
References
[1] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong. Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-9948, 2020.
[2] Xiaoyang Wu, Yixing Lao, Li Jiang. Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based based pooling. In Advances in Neural Information Processing Systems, pages 33330-33342, 2022.
[3] Bowen Cheng. Maxwell D Collins, Yukun Zhu Ting Liu, Thomas S Huang, Hartwig Adam, and Liang- Chich Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-u panoptic segmentation. In IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 12475-12488 2020.
[4] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[5] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yucxin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IÉEE/CVF Conference Computer Vision and Pattern Recognition, pages 9939-9948. 2021.
[6] Yuenan Hou, Xinge Zhu, Yuexin Ma. Chen Change Loy, and Yikang Li. Point-to-voxel knowledge distillation for lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8479-8488, 2022.
[7] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpynet: A deep and efficient peeee-point-ooxe fusion network for lidar point cloud segmentation. In IEEE/CVF International Conference on Computer Vision, pages 16024-16033. 2021.
[8] Ran Cheng. Ryan Razani, Ehsan Taghavi, Eaxu Li, and Bingbing Liu. (af)2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12547-12556. 2021.
[9] Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li. Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In IEEE/CVF International Conference Computer Vision, pages 228 228-2400222 2023.
10

---

[10] Fangzhou Hong, Lingdong Kong, Hui Zhou. Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3d and 4d panoptic segmentation via dynamic shifting network. arXiv preprini arXiv:2203.07186 2022.
[11] Runnan Chen, Xinge Zhu, Nenglun Chen, Wei Li, Yuexin Ma, Ruigang Yang. and Wenping Wang. Zero- shot point cloud segmentation by transferring geometric primitives. arXiv preprint arXiv:2210.09923, 2022.
[12] Bjöm Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In International Conference on 3D Vision, pages 2021.
[13] Yueh-Cheng Liu, Ku-Kai Huang, Hung- Yueh Chiang. Hung- Ting Su, Zhe- Yu Liu, Chin-Tang Chen, Ching- Yu Tseng, and Winston H Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint arXiv:2 2021.
[14] Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9891-9901, 2022.
[15] Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11705-21715 2023.
[16] Lingdong Kong. Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEFE International Conference on Robotics and Automation, pages 9338 994889945,559 2023.
[17] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao. and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020-7030 2023.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[19] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[20] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages (96-712, 2022.
[21] Yiteng Xu. Peishan Cong, Yichen Yao, Runnan Chen, Yuenan Hou, Xinge Zhu, Xuming He. Jingyi Yu, and Yuexin Ma. Human-centric scene understanding for 3d large-scale scenarios. In IEEE/CVF International Conference on Computer Vision, pages 20349-20359, 2023.
[22] Youquan Liu, Runnan Chen, Xin Li, Lingdong Kong, Yuchen Yang. Zhaoyang Xia, Yeqi Bai, Xinge Zhu, Yuexin Ma, Yikang Li, et al. Uniseg: A unified multi-modal lidar segmentation network and the openpeseg codebase. In IEEE/CVF International Conference on Computer Vision. pages 21662-2111 21673, 2023.
[23] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:1 14:178644 -17744477775, 2021.
[24] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In IEEE/CVF International Conference on Computer Vision, pages 7262-7272 2021.
[25] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[26] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In IEEE/CVF International Conference on Computer Vision, pages 12179-12188, 2021.
[27] Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li. et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14408 14419, 2022.
11

---

[28]
Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmenta- tion. arXiv preprint arXiv:2005. 10821, 2020.
[29] Runnan Chen. Studies on attention modeling for visual understanding. HKU Theses Online, 2023.
[30]
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1686611695, 2022.
[31] Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu, Roho3d: Towards robust and reliable 3d perception against corruptions. JEEE/CVF International Conference on Computer Vision, pages 19994-2000, 2023.
[32] Runnan Chen, Penghao Zhou, Wenzhe Wang. Nenglun Chen, Pai Peng, Xing Sun, and Wenping Wang- Pr-net: Preference reasoning for personalized video highlight detection. In IEEE/CVF International Conference on Computer Vision, pages 7980-7989, 2021.
[33]
MMDetectionn3D Contributors. Mmdetcction3d: Openmmlab next-generation platform for general 3d object detection, 2020.
[34] Lingdong Kong. Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, I Liang Pan, Kai Chen, and Ziwei Liu. Benchmarking 3d perception robustness common corruptions and sensor failure. In International Conference on Learning Representations Workshops, 2023.
[35] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. In Advances in Neural Information Processing Systems. 2023.
[36] Runnan Chen, Xinge Zhu, Nenglun Chen, Dawei Wang, Wei Li, Yuexin Ma, Ruigang Yang. Tongliang Liu, and Wenping Wang. Model2scene: Learning 3d scene representation via contrastive language-cad models pre- wre-trainigg. arXiv preprint arXiv:2309.16956 2023.
[37] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any- view self-supervised object segmentation complex scenes. In International Conference on Learning Representations, 2022.
[38] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[39]
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sule-supe learning. In Advances in Neural Information Processing 2020.
Systems, volume 33, pages
[40] Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems, volume 27, 2014.
[41] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. JEEE/CVF International Conference on Computer Vision, pages 1422-144 1430, 2015.
[42] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15750-15755 2021.
[43] Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607, 2020.
[44] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulia. Unsuper- vised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912-9924. 2020.
[45] Runnan Chen, Xinge Zhu, Nenglun Chen, Dawei Wang, Wei Li, Yuexin Ma, Ruigang Yang, and Wenping Wang. Towards 3d scene understanding by referring synthetic models, arXiv preprint arXiv:2203. 10546. 2022.
[46] Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsupervised learning of intrinsic structural representation points. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9121-9130, 2020.
12

---

[47] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 2613-2622, 2021.
[48] Yuchao Wang. Hanchen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. /EEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4248-4257, 2022.
[49] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi semi-super semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4268 00084447,220 2022.
[50] Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 14:22106-22118, 2021.
[51] Xiangde Luo, Minhao Hu, Tao Song, Guotai Wang, and Shaoting Zhang. Semi-supervinn medical image segmentation via cross teaching hetween enn and transformer. International Conference on Medical Imaging with Deep Learning, pages 820-833. PMLR, 2022.
[52] Wei Shen, Zelin Peng, Xuehui Wang, Huayu Wang. Jiazhong Cen, Dongsheng Jiang, Lingxi Xie, Xiaokang Yang, and Q Tian. A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):9284 9305, 2023.
[53] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12674-12684, 2020.
[54] Runnan Chen, Yuexin Ma, Lingjie Liu, Nenglun Chen, Zhiming Cui, Guodong Wei, and Wenping Wang. Bemi-supervised anatomical landmark detection via chape-regulated self- self-training. Neurocomputing. 471 :3:5-333355, 2022.
[55] Yuhang Lu, Qi Jiang, Runnan Chen, Yuenan Hou, Xinge Zhu, and Yuexin Ma. See more and know more: Zero-shot point cloud segmentation via multi-modal visual data. In IEEE/CVF International Conference on Computer Vision, pages 21674-21668 21644, 2023.
[56] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211. 2022.
[S7] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language-driven open- vpen-vocabulary 3d seene understanding. arXiv preprint ar arxiv:2211. 16312, 2022.
[58] Laigi Riz, Cristiano Saltori, Elisa Ricci, and Fabio Poiesi. Novel class discovery for 3d point cloud semantic segmentation. arXiv preprint arXiv:2303. / 2023.
[59] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. arXiv preprint 12.1:271 2021.
[60] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference Learning Representations, 2022.
[61] Maxime Bucher, Tuan- Vu, Matthiuu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[62] Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmentation. Advances in Neural Information Processing Systems, 33: 13317-13733. 2020.
[63] Ping Hu, Stan Sclaroff, and Kate Saenko. Uncertainty aware learning for zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 33:21713 21724, 2020.
[64] Hui Zhang and Henghui Ding. Prototypical matching and open set rejection for zero-shot semantic segmentation. In JEEE/CVF International Conference on Computer Vision, pages 6974-6983, 2021.
[65] Xu Yan, Jiantao Gao, Chaoda Zheng, Chaoda Zheng, Ruimao Zhang, Shenghui Cui, and Zhen Li. 2dpass:
2d priors assisted semantic segmentation on lidar point clouds. In European Conference on Computer Vision, pages 677-695, 2022.
13

---

[66] Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar sell-supervised distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9891-9901, 2022.
[67] Xidong Peng, Runnan Chen, Feng Qiao, Lingdong Kong, Youquan Liu, Tai Wang, Xinge Zhu, and Yuexin Ma. Sam-guided unsupervised domain adaptation for 3d segmentation. arXiv preprini arXiv:2310.08820
2023.
[68] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to lahel noise: A loss correction approach. In IEEE/CVE Conference on Computer Vision and Pattern Recognition. pages 1944-1999 1952, 2017.
[69] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning, pages 1789-1799, PMLR, 2020.
[70] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3):447 461, 2015.
[71] Xiaobo Xia. Tongliang Liu, Nannan Wang. Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32, 2019.
[72] Brendan Van Rooyen and Robert C Williamson. A theory of learning with corrupted labels. J. Mach. Learn. Res.,
2017.
[73] Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu. Estimating noise transition matrix with label correlations for noisy multi-label learning. In Advances in Neural Information Processing Systems, 2022.
(74] Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance- dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning, pages 25302-25312. PMLR, 2022.
[75] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems, 26, 2013.
[76] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Reprexentations, 2020.
[77] Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly supervised learning and label noise robustness. In International Conference on Machine Learning, pages 708-717. PMLR, 2016.
[78] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[79] Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances in Neural Information Processing Systems, 28, 2015.
[80] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. -izeiindependent sample complexity of neural networks. Conference On Learning Theory, pages 297-299. PMLR, 2018
[81] Florian Jell. Theoretical foundations. In Patent Filing Strategies and Patent Management: An Empirical Study, pages 4 14. Springer, 2012.
[82] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems, 31, 2018.
[83] Mingehen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with carly stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statisticx, pages 4313-4324. PMLR, 2020.
[84] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 6473):007-155, 2021.
[85] Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pages 33-2442. PMLR, 2017.
14

---

[86] Yingbin Bai, Erkun Yang. Zhaoqing Wang. Yuxuan Du, Bo Han. Cheng Deng, Dadong Wang, and Tongliang Liu. Msr: Making self-supervised learning robust aggressive augmentations. arXiv preprint arXiv:2206. 01999, 2022.
[87] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255, 2021.
[88] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 770-778, 2016.
[89] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Europeun Conference on Computer Vision, pages 219-229, 2020.
[90] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[91] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-an Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5828 5839, 2017.
[92] Holger Cacsar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11621-11631, 2020.
[93] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Reijhom. nuscenes: A multimodal dataset for autonomous driving. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11621-11631, 2020.
[94] Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters, 7:3795-3802, 2022.
[95] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[96] Liang-Chiel Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 1706. 2017.
15


Reviews:
[{"review_id": "Ir9FsZU06P", "reviewer": "Reviewer_9W6n", "summary": "The paper proposes a new method for jointly learning 2D and 3D segmentation models that can be queried with open-world text queries. The paper's main contribution is a new Cross-modality Noisy Supervision to reduce the influence of noise on the predictions. The proposed method uses a SAM model to predict masks for every pixel and then uses those noises to regularize CLIP's noisy dense pixel features. The results shown in the paper show that the proposed method outperforms baselines in both 2D and 3D segmentation. I have a couple of questions and suggestions to improve the paper and I am willing to increase my score if these are sufficiently addressed in the rebutal. ", "questions": "I have a few clarification questions as well:\n\n1. Line 159: What is meant by pixel-point calibration? Are the camera poses not known? Is there an additional step required apart from simple projection? If not, this section can be shortened in my opinion.\n\n2. Line 160: What is meant by the domain gap here? And why does this matter in the first place? As I see it, the method tries to align the features from the 3D network to the 2D CLIP/SAM space, and the domain gap should not matter here, right?\n\n3. Line 209: In the current method, you use cosine similarity to force 2D/3D network outputs to be similar to the SAM feature output. Is this objective not in contrast to enforcing similarity between CLIP features and the 3D/3D network outputs? Especially when looking at the results in Table 2, this specific regularization does not seem to add a lot in terms of performance (Latent Space Consistency Regularization)", "limitations": "The paper does not discuss limitations and broader impact in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well-written and mostly easy to follow (apart from the mathematical notation, see comments below). The paper provides a pretty comprehensive overview of the literature up until CVPR 2023, and I also liked the qualitative results and Figure 1, which was informative. ", "weaknesses": "I have some suggestions (in no particular order) to improve the paper:\n\n1. I take some issue with the claim that the method is the first to combine SAM and CLIP for label-free scene understanding. Multiple different methods have combined 2D segmentation models with CLIP, and as these systems are built modularity, they have already been able to switch out their segmenters for SAM. Two papers that come to mind for this are OvSeg (2D segmentation) and ClipFusion (3D segmentation). I suggest that this claim be softened.\n\n2. Figure 2 is very non-intuitive in my opinion and I did not fully understand the flow of information even after multiple reads. Maybe it would be good to improve this figure.\n\n3. Section 3.2 seems to be overly mathy and the notation is extremely convoluted. Consider cleaning up the super/sub scripts, and I also have a few additional comments:\n* Equations 2 and 3 take a lot of space do not add anything to my understanding of the method. I suggest removing them. \n* It is not necessary to define the range of each variable. \n* Equations 5/6 are extremely convoluted. Consider simpler notation.\n* In equation 1, I think there is an error, and the dot product should be between c_i^T, t_l\n\n4. Regarding the results in Table 1, the OpenScene results seem to be much lower compared to the ones reported in the original paper. How can this be explained? Was the baseline retrained for this paper?\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 9W6n for devoting time to this review and providing valuable comments.\n\n---\n> ***Weaknesses 1:** \"The claim that the method is the first to combine SAM and CLIP.\"*\n\n**A:** Thanks for your suggestion! Considering previous methods can easily replace their 2D network with SAM, we will modify the unrigorous claim \"the first work to combine SAM and CLIP\". Instead, we would like to mention that we study how CLIP and SAM enable networks to comprehend 2D and 3D environments without relying on labelled data.\n\n---\n> ***Weaknesses 2:** \"Figure 2 is very non-intuitive.\"*\n\n**A:** Thanks for your valuable advice! As shown in the supplement PDF file, we have improved Figure 2 to demonstrate the framework better. Specifically, we visualize the intermedia results of network predictions. Besides, we detailed the process of CLIP's pseudo-labeling and label refinement (on the right-hand side). Please let us know if there is anything unclear. \n\n---\n> ***Weaknesses 3:** \"Section 3.2 seems to be overly mathy and the notation is extremely convoluted.\"*\n\n**A:** We will correct Equations 1, remove Equations 2 and 3, and clean the variable range (super/sub-scripts). We will simplify the loss function (Equations 5/6). For example Equations 5, $\\mathcal{L}_{s1} = \\sum\\_{i} CE(x_i, P^*\\_{x_i}) + CE(p_i, P^*\\_{p_i})$, where $CE(x, y) =- \\log\\frac{\\exp(\\psi(\\theta\\_s(x), t\\_{y}))}{\\sum\\_{l}\\exp(\\psi(\\theta\\_s(x), t\\_{l}))}$. Thanks!\n\n---\n> ***Weaknesses 4:** \"Regarding the results in Table 1.\"*\n\n**A:** OpenScene adapts the OpenSeg and LSeg as the 2D backbone, where OpenSeg and LSeg are trained on labelled images and significantly outperform our 2D backbone MaskCLIP. Therefore, for a fair comparison in the label-free setting, we replace OpenScene's 2D backbone with MaskCLIP, which is without training on any labelled images. To this end, OpenScene achieves much lower performance than that reported in the original paper.\n\n---\n> ***Questions 1:** \"Line 159: What is meant by pixel-point calibration? Are the camera poses not known? Is there an additional step required apart from simple projection?\"*\n\n**A:** Pixel-point calibration means pixel-point correspondence by 2D image and 3D point cloud calibration. Note that the dataset already provides the camera poses, so we obtain the pixel-point correspondence via simple projection. We have fixed the ambiguous expression in the revised version. Thanks!\n\n---\n> ***Questions 2:** \"Line 160: What is meant by the domain gap here? And why does this matter in the first place?\"*\n\n**A:** The domain gap means that 2D images and 3D point clouds have different representations. Therefore, the vision foundation models, which are trained on 2D images, could not be directly used to process 3D point clouds. To this end, we transfer the vision foundation models' knowledge from 2D images to 3D point clouds via dense pixel-point correspondence. We have improved the presentation in the revised version. Thanks!\n\n---\n> ***Questions 3:** \"Line 209: You use cosine similarity to force 2D/3D network outputs to be similar to the SAM feature output. Is this objective not in contrast to enforcing similarity between CLIP features and the 3D/3D network outputs?\"*\n\n**A:** Thanks for the comment.\n- Good question! We have encountered this issue and found that the optimization conflict reduces performance. To solve this issue, we adopt a simple yet effective solution in our paper. Specifically, we use different linear mapping layers in Latent Space Consistency Regularization and Prediction Consistency Regularization. That is, the 2D and 3D features are respectively mapped to other feature spaces by $\\theta_{f}$ and $\\varphi_{f}$ (lines 222)before aligning with the SAM feature. Meanwhile, 2D and 3D features are respectively mapped by $\\theta_s$ and $\\varphi_s$ (lines 193) before aligning with the CLIP's text embedding. Therefore, these two optimizations are not directly conflict with each other. \n\n- The performance gain in Latent Space Consistency Regularization seems marginal because it is conducted on our full method. Therefore, we conduct the following ablation experiments, which may verify the effectiveness better. Firstly, we impose the baseline method (lines 272) self-training with the CLIP pseudo-labels, termed \"baseline + CLIP\". Secondly, building upon \"baseline + CLIP\", we additionally impose Latent Space Consistency Regularization, termed \"baseline + CLIP + LSCR\". The improvements from \"baseline + CLIP\" to \"baseline + CLIP + LSCR\" are 2.1 and 4.9 mIoU in ScanNet 2D and 3D datasets, respectively, which is more prominent.\n|Method|ScanNet 2D|ScanNet 3D|\n|-|:-:|:-:\n|baseline|17.3|14.2\n|baseline + CLIP|20.4|23.8\n|baseline + CLIP + LSCR|22.5|28.7\n|Full method|28.4|33.5\n\n- Besides, as shown in the supplement pdf file, we visualize the feature space with (w LSCR) or without (w/o LSCR) latent space regularization by using T-SNE (t-distributed Stochastic Neighbor Embedding) [P1]. T-SNE is an unsupervised non-linear dimensionality reduction technique for data exploration and visualizing high-dimensional data. By our latent space regularization, for those pixels/points with the same semantics (the same colour), their feature space tends to be more compact. Meanwhile, for those pixels/points with different semantics (the different colours), their feature space tends to be more distinguishable.\n \n---\n> ***Limitations:** \"The paper does not discuss limitations and broader impact in the paper.\"*\n\n**A:** Limited by space, we discussed the limitation and future work in supplementary materials. As for the broader impacts, we believe our method is meaningful for the artificial general intelligence field because it bridges the vision and language, percept real-world environments without relying on any labelled data. We hope our work will inspire and motivate further research in this area.\n\n---\n**References:**\n- [P1] L. Van der Maaten and G. Hinton. Visualizing High-Dimensional Data Using t-SNE. JMLR, 2008.\n"}, {"review_id": "Pvpzx6C68I", "reviewer": "Reviewer_Mxk9", "summary": "This paper proposes using SAM to improve the quality of segmentation masks generated by CLIPs. The refined masks are then used as pseudo-labels to train networks that excel at segmentation tasks, following a similar training process to MaskCLIP+. Based on this idea, a two-stage self-training process is proposed to simultaneously train 2D and 3D networks with SAM-refined masks and self-generated masks iteratively. To mitigate mask noise, the authors further propose latent space consistency regularization to transfer knowledge from the SAM feature space to segmentation networks. The final trained segmentation networks demonstrate impressive performance on both 2D and 3D segmentation datasets.", "questions": "Please refer to Weaknesses.", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "(1) The overall method is both simple and effective. Although the authors use SAM trained with labeled data, which means it is not strictly label-free, I believe that it is acceptable given the performance improvement and contribution to the community.\n\n(2) The performance is impressively prefect.\n\n(3) The ablation study proves the gains of the main contributions of the paper.\n\n(4) The paper is well-written and easy to read and understand except for some minor issues.\n", "weaknesses": "(1) In this paper, the authors chose DeeplabV3 as the 2D backbone and MinkowskiNet34 as the 3D backbone. However, previous works such as MaskCLIP used DeeplabV2 as the 2D backbone and CLIP2Scene used MinkowskiNet14 as the 3D backbone. It is obviously that the backbones used in this paper have better performance, so it may not be entirely fair to compare them with previous works.\n\n(2) I'm a bit confused about the details of the Label Refinement by SAM. SAM often outputs very fine-grained masks and produces multi-level masks, while CLIP also often outputs masks with a lot of discrete noise points. How to handle these situations? Can the author provide a simple pseudo-code to describe this part of the processing?\n\n(3) I think the baseline selected by this paper is not appropriate. While the main contributions of this paper are the improvement of pseudo-labeling quality and more effective training strategies, the baseline should be a basic method that incorporates both of these points. For example, I think that for 2D scenes, MaskCLIP+ is a better baseline than CLIP. \n\n(4) I think the presentation of Table 2 could be improved. Except for the experimental settings in the first row, the configurations in the other rows are abridged versions of the full configuration in a specific structure. To better present the impact of each structure on the final configuration, I suggest comparing the results of each ablation experiment with the performance drop of the full configuration results.\n\n(5) MaskCLIP++ -> MaskCLIP+.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Mxk9 for devoting time to this review and providing valuable comments.\n\n---\n> ***Weaknesses 1:** \"the authors chose DeeplabV3 as the 2D backbone and MinkowskiNet34 as the 3D backbone. However, previous works such as MaskCLIP used DeeplabV2 as the 2D backbone and CLIP2Scene used MinkowskiNet14 as the 3D backbone.\"*\n\n**A:** Thanks for your detailed comments. We would like to claim that we re-implement MaskCLIP for comparison. Therefore, the re-implement MaskCLIP shares the same 2D backbone (DeeplabV3) as well as the same data augmentations with our method. As for the comparison with CLIP2Scene, we conduct the experiment that adapts MinkowskiNet14 as the 3D backbone for our method. We achieve 27.6, 32.3, and 25.2 mIoU in ScanNet 2D, 3D, and nuScenes datasets, which still significantly outperform CLIP2Scene. We have supplemented more experiment details in the revised version. Thanks!\n\n|Method|ScanNet 2D|ScanNet 3D|nuScenes\n|-|:-:|:-:|:-:\n|CLIP2Scene|23.7|25.6|20.8\n|Ours (MinkowskiNet14)|27.6|32.3|25.2\n|Ours|28.4|33.5|26.8\n\n---\n> ***Weaknesses 2:** \"I'm a bit confused about the details of the Label Refinement by SAM\"*\n\n**A:** We follow the official implementation of SAM that produces object masks and remove the redundant masks. Next, we enumerate each mask and adjust the pseudo-labels to be identical within the same mask. Specifically, we adopt the max voting strategy that unifies all pixels' labels to be the semantic labels that most pixels belong to. Our supplementary materials provide the source code in the function \"label_refinement_sam\" of the file: \"codes/pretrain/lightning_trainer.py\". By the way, we visualize the qualitative results of the Label Refinement by SAM in Figures 2 and 3 and the Figures in the supplementary materials, where \"Ours w/o CNS\" indicates the results of CLIP's label Refinement by SAM. Besides, we present the following pseudo-code for better understanding.\n\n      Algorithm: label_refinement_sam\n\n      Input: pseudo-labels (H * W), masks_sams (N * H * W) \n\n      (H and W indicate the height and width of an image. N is the number of masks. A mask consists of 0 or 1.)\n\n      Process:\n\n      for i, mask in enumerate(masks_sam):\n\n           pseudo-labels[mask] = argmax(pseudo-labels[mask])\n\n      Return: pseudo-labels\n\n> ***Weaknesses 3:** \"I think the baseline selected by this paper is not appropriate.\"*\n\n**A:** Thanks for your suggestion. We have adopted MaskCLIP+ as the 2D baseline in the revised version.\n\n> ***Weaknesses 4:** \"I think the presentation of Table 2 could be improved.\"*\n\n**A:** Good points! We will follow your suggestion in the revised version. Thanks!\n\n> ***Weaknesses 5:** \"MaskCLIP++ -> MaskCLIP+.\"*\n\n**A:** Thanks for pointing out the typo. We have fixed it in the revised version.\n"}, {"review_id": "oSw5LzLOch", "reviewer": "Reviewer_rM41", "summary": "This work employs 2D segmentation foundation models to perform semantic segmentation for both 2D and 3D indoor scenes. The clip model provides a general understanding of the semantic content in each image, while the SAM model generates precise segmentation masks. By integrating these two models, this study demonstrates improved quantitative results on two datasets.", "questions": "As the reviewer's question mainly lies in the clarity and writing, please refer to the \"Weakness\" section.", "limitations": "Please refer to the \"Weakness\" section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The significant enhancement in segmentation results is achieved by leveraging two distinctively trained 2D segmentation models, SAM and Clip, surpassing the previous approach that solely utilized the Clip model.\n\n2. It is reasonable to anticipate improved segmentation outcomes by combining two powerful models that provide both semantic understanding and precise mask contours. The merging of diverse masks through pooling is straightforward and appears to be the most effective element contributing to the performance gain.", "weaknesses": "1. Writing: The authors are encouraged to provide further clarification on the \"calibration matrix,\" as it serves as the initial step in connecting 2D pixels and 3D points. How was this connection established without knowledge of the depth of each pixel? Additionally, could you please clarify the meaning of \"Image Number\" mentioned in line 305 and indicate the corresponding table or figure for reference? A similar question arises regarding the ablation study mentioned in line 290.\n\n2. Clarity:  In the abstract, the authors claim that the proposed method outperforms the state-of-the-art by a significant margin, but the method's name is not mentioned. When the term \"calibration matrix\" is first introduced in line 51, could you please provide a clear explanation? Does the proposed method predict both 2D and 3D semantic label maps? In Figure 2, which model requires training/finetuning and which one is frozen? How is the SAM prompted to predict the segmentation maps?\n\n3. As a straightforward baseline, the authors could utilize Clip to generate coarse semantic masks and merge these masks with SAM segmentation results. It is recommended to present the performance in Table 2 and the corresponding figure. The suggested experiments should be evaluated on the large-scale Scannet dataset. Furthermore, if the 2D foundation models already perform well, the motivation behind the \"Prediction Consistency Regularization\" is not clear. Why is there a need to fine-tune your own model?\n\n4. Although the demonstrated segmentation results are close to the ground truth, the mIoU scores for both 2D and 3D remain below 35.\n\n5. The improvement achieved through \"Latent Space Consistency Regularization\" appears to be minimal for both 2D and 3D segmentation, despite being listed as an important contribution.\n\n6. Please include citations for NeRF-based and mesh-based scene segmentation:\n\n[1] Atlas: End-to-end 3d scene reconstruction from posed images (supervised and unsupervised)\n[2] Decomposing nerf for editing via feature field distillation\n[3] Nerf-sos: Any-view self-supervised object segmentation on complex scenes \n[4] Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer rM41 for devoting time to this review and providing valuable comments.\n\n---\n> ***Weaknesses 1:** \"Writing: The authors are encouraged to provide further clarification on the \"calibration matrix,\" as it serves as the initial step in connecting 2D pixels and 3D points.\"*\n\n**A:** Thanks for your detailed comments! We intend to express that we transfer the pseudo-label from 2D image pixels to 3D points using pixel-point correspondence. Pixel-point correspondence is obtained by direct projection between the 2D images and the 3D point cloud, where the pixel depth and camera poses are provided in the dataset. We have improved the presentation in the revised version. Thanks!\n\n\n---\n> ***Weaknesses 2:** \"2. Clarity: In the abstract, the authors claim that the proposed method outperforms the state-of-the-art by a significant margin, but the method's name is not mentioned. When the term \"calibration matrix\" is first introduced in line 51, could you please provide a clear explanation? Does the proposed method predict both 2D and 3D semantic label maps? In Figure 2, which model requires training/finetuning, and which one is frozen? How is the SAM prompted to predict the segmentation maps?\"*\n\n**A:** \n- Thanks for your comments. We will acknowledge the state-of-the-art CLIP2Scene in the abstract. \n- The calibration matrix here means the pixel-point correspondence that transfers pseudo-label from 2D image pixels to 3D points. We have improved the presentation in the revised version. \n- Our method is designed for label-free 2D and 3D scene understanding, so it predicts both 2D and 3D semantic label maps (lines 16 and 76). \n- We have improved Figure 2. As shown in the supplement pdf file, 2D and 3D networks are needed for training, while SAM and CLIP are frozen during training.\n- Following the official implementation of SAM, we use the function \"SamAutomaticMaskGenerator\" to produce the masks. Basically, it first uniform samples the seed points in an image, then cluster those pixels with similar features to the seed points. As a result, each cluster can be regarded as a mask for the images. We have supplemented more details in the revised version. Thanks!\n\n---\n> ***Weaknesses 3:** \"It is recommended to present the performance in Table 2 and the corresponding figure. The suggested experiments should be evaluated on the large-scale Scannet dataset. Furthermore, if the 2D foundation models already perform well, the motivation behind the \"Prediction Consistency Regularization\" is not clear. Why is there a need to fine-tune your own model?\"*\n\n**A:** Thanks for your comments. We would like to clarify that we have conducted such experiments in the paper, termed \"Ours w/o CNS\". The qualitative and quantitative results are shown in Figures 3, 4, and Table 2. And more 2D and 3D qualitative results are shown in the supplementary materials. Note that all the experiments are evaluated on 25K 2D images and 1513 3D scans in the ScanNet dataset (lines 239), which is the most comprehensive dataset to our knowledge. Besides, we analyze the results in the ablation section (lines 275). As shown in Figures 3, 4, and Table 2, \"Ours w/o CNS\" achieves sub-optimal performance. The merged CLIP and SAM results are unsatisfactory and still suffer from noise prediction. The motivation behind the \"Prediction Consistency Regularization\" is that different networks have merits in different aspects. Co-training [P1] the different networks helps to mitigate error propagation and prevent the networks from overfitting the noisy labels (lines 53). We have clarified the above details in the revised version. Thanks!\n\n---\n> ***Weaknesses 4:** \"4.Although the demonstrated segmentation results are close to the ground truth, the mIoU scores for both 2D and 3D remain below 35.\"*\n\n**A:** Label-free scene understanding is a very challenging task. We show some representative cases and hope to inspire the research community. Besides, there are also some failure cases in the supplementary materials, especially in the nuScenes dataset. Thanks!\n\n---\n> ***Weaknesses 5:** \"The improvement achieved through \"Latent Space Consistency Regularization\" appears to be minimal for both 2D and 3D segmentation.\"*\n\n**A:** \n- The performance gain in Latent Space Consistency Regularization seems marginal because it is conducted on our full method. Therefore, we conduct the following ablation experiments, which may verify the effectiveness better. Firstly, we impose the baseline method (lines 272) self-training with the CLIP pseudo-labels, termed \"baseline + CLIP\". Secondly, building upon \"baseline + CLIP\", we additionally impose Latent Space Consistency Regularization, termed \"baseline + CLIP + LSCR\". The improvements from \"baseline + CLIP\" to \"baseline + CLIP + LSCR\" are 2.1 and 4.9 mIoU in ScanNet 2D and 3D datasets, respectively, which is more prominent.\n\n|Method|ScanNet 2D|ScanNet 3D|\n|-|:-:|:-:\n|baseline|17.3|14.2\n|baseline + CLIP|20.4|23.8\n|baseline + CLIP + LSCR|22.5|28.7\n|Full method|28.4|33.5\n\n- Besides, as shown in the supplement pdf file, we visualize the feature space with (w LSCR) or without (w/o LSCR) latent space regularization by using T-SNE (t-distributed Stochastic Neighbor Embedding) [P1]. T-SNE is an unsupervised non-linear dimensionality reduction technique for data exploration and visualizing high-dimensional data. By our latent space regularization, for those pixels/points with the same semantics (the same colour), their feature space tends to be more compact. Meanwhile, for those pixels/points with different semantics (the different colours), their feature space tends to be more distinguishable.\n\n---\n> ***Weaknesses 6:** \"Please include citations for NeRF-based and mesh-based scene segmentation.\"*\n\n**A:** We will investigate and discuss the methods in the revised version. Thanks!\n\n---\n**References:**\n- [P1] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels\n"}, {"review_id": "XZETnoyb04", "reviewer": "Reviewer_inkR", "summary": "The paper proposes a method that utilizes the SAM and CLIP (Contrastive Language-Image Pretraining) models for understanding the 2D and 3D world in a label-free manner. The authors suggest a two-step process to improve the results obtained from CLIP.\n\nIn the first step, the authors generate a noisy output from the CLIP model, which is primarily designed for semantic tasks. This output may contain imperfections or inaccuracies due to the inherent limitations of CLIP.To refine the output and obtain more visually appealing semantic results, the authors employ a SAM network in the second step. The SAM network is designed to enhance the quality of the semantic results generated by CLIP.\n\nFurthermore, the paper introduces a Cross-modality Noise Supervision module, which aims to optimize the training of both the 2D and 3D models simultaneously. \nThe authors demonstrate promising results on datasets such as ScanNet and nuScenes. These datasets are commonly used in the field of computer vision and provide challenging scenarios for understanding the 2D and 3D world. The encouraging results obtained from these datasets suggest the effectiveness of the proposed approach in understanding and representing the visual and semantic aspects of the real world.", "questions": "1. Could the authors provide intermediate feature space visualization which shows comparative results with and without latent space regularization ? Since from the ablation experiments provided there seems to be marginal improvement in mIoU numbers both in the ScanNet and nuScenes dataset.", "limitations": "1. The method proposed assumes SAM to be perfect in sense of providing meaning masks and does not train the SAM model. But fails to give examples or quote scenarios where this might fail, considering SAM outputs a mask which is not meaningful or helping to noisy CLIP based labels.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strength:\n\n1. Paper pushes the current boundary of training models with label free training data, using pre-existing models which have shown good performance on zero-shot tasks.\n2. Proposes to seamlessly refine labels predicted by CLIP model to achieve good results in Indoor Dataset.\n3. Paper for most of the part is well written and has a definite flow to it.\n4. Beats existing method such MaskCLIP, CLIP2scene models comprehensively to get state-of-arts results.", "weaknesses": "1. Missing Ablation: Prediction Consistency Regularization, the second stage where there is random switching of pseudo labels between 2D, 3D and CLIP based intermediate outputs, is not well explained. \n\n2. Missing Ablation: Ablation study which shows what is the impact of this random switching with and without. We also need to see what if we only employ one of the three pseudo labels and not all three, what impact does that have. Seeing the impact of each individual is necessary to comment about its efficacy.\n\n3. Effect of Latent Space Consistency Regularization: There is only marginal improvement seen using SAM features to guide the feature of intermediate 2D/3D encoder features. Do the authors have intermediate results(qualitative), which justifies how SAM features actually help the 2D and 3D features ?\n\nTypos:\n1. Line no 169: is the doc product operation. I presume this is supposed to be dot product.\n2. In Fig:2, How do we achieve the Refined CLIP Pseudo Label and how is this different from Refined 2D Pseudo labels. Please add further information for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer inkR for devoting time to this review and providing valuable comments.\n\n---\n> ***Weaknesses 1:** \"Random switching of pseudo labels is not well explained.\"*\n\n**A:** Thanks! We would like to explain how we conducted ablation experiments about label random switching. The ablation experiment settings are in the paragraph \"Effect of Prediction Consistency Regularization\" (lines 296-304), and we have elaborated this aspect in a more straightforward way in the revised version:\n\n1) In our full method, both the 2D and 3D network's labels were randomly switched among CLIP, 2D and 3D prediction.\n2) We experimented with the setting \"w/o CT\". In this experiment, the 2D and 3D networks were not allowed to cross-train each other in the second training stage, i.e., the 2D network's labels were randomly switched between CLIP and 2D prediction. Meanwhile, the 3D network's labels were randomly switched between CLIP and 3D prediction.\n3) We experimented \"w/o SCT\". This experiment eliminates self- and cross-training in 2D and 3D networks, i.e., 2D and 3D network labels are derived solely from CLIP.\n4) The experiment \"w/o CLIP\", this experiment excludes the usage of CLIP's labels during the second training stage, i.e., both the 2D and 3D network's labels were randomly switched between 2D and 3D prediction. \n\nAs shown in Table 2, our full method achieves the best performance. We also observe that the method fails when CLIP's supervision is absent during the second training stage. This failure can be attributed to the tendency of the 2D and 3D networks to \"co-corrupt\" each other with extremely noisy labels.\n\n---\n> ***Weaknesses 2:** \"Missing Ablation: Ablation study which shows what is the impact of this random switching with and without.\"*\n\n**A:** Thanks for your suggestions! We additionally conduct the following ablation experiments to see the impact of random switching with and without.\n1) \"W CLIP\" (the same as \"w/o SCT\"): employing only CLIP's prediction to train 2D and 3D networks. The mIoU is 24.7 and 30.6 in 2D and 3D, respectively.\n2) \"W 2D\": employing only 2D network's prediction to train 2D and 3D networks. The mIoU is 6.9 and 8.8 in 2D and 3D, respectively.\n3) \"W 3D\": employing only 3D network's prediction to train 2D and 3D networks. The mIoU is 4.8 and 6.4 in 2D and 3D, respectively.\n\n\n|Method|ScanNet 2D|ScanNet 3D|\n|-|:-:|:-:\n|W CLIP|24.7|30.6\n|W 2D|6.9|8.8\n|W 3D|4.8|6.4\n|Full method|28.4|33.5\n\nThe experiments show the effectiveness of employing the random switching strategy, and the 2D and 3D tend to corrupt if without the guidance of CLIP's pseudo labels. We have supplemented the above experiments in the revised version. Thanks!\n\n---\n> ***Weaknesses 3:** \"Effect of Latent Space Consistency Regularization.\"*\n\n**A:** - The performance gain in Latent Space Consistency Regularization seems marginal because it is conducted on our full method. Therefore, we conduct the following ablation experiments, which may verify the effectiveness better. Firstly, we impose the baseline method (lines 272) self-training with the CLIP pseudo-labels, termed \"baseline + CLIP\". Secondly, building upon \"baseline + CLIP\", we additionally impose Latent Space Consistency Regularization, termed \"baseline + CLIP + LSCR\". The improvements from \"baseline + CLIP\" to \"baseline + CLIP + LSCR\" are 2.1 and 4.9 mIoU in ScanNet 2D and 3D datasets, respectively, which is more prominent.\n|Method|ScanNet 2D|ScanNet 3D|\n|-|:-:|:-:\n|baseline|17.3|14.2\n|baseline + CLIP|20.4|23.8\n|baseline + CLIP + LSCR|22.5|28.7\n|Full method|28.4|33.5\n\n---\n> ***Typos 1:** \"Line no 169: is the doc product operation. I presume this is supposed to be dot product.\"*\n\n**A:** Thanks for pointing out the typos, we have fixed it in the revised version.\n\n---\n> ***Typos 2:** \"How do we achieve the Refined CLIP Pseudo Label and how is this different from Refined 2D Pseudo labels.\"*\n\n**A:** Thanks for your suggestions! Please check our improved figure in the supplement PDF file. Specifically, we detailed the process of CLIP's pseudo-labeling and label refinement (on the right-hand side). We hope it is better for understanding our method. Please let us know if there is anything unclear.\n\n\n---\n> ***Questions 1:** \"Could the authors provide intermediate feature space visualization which shows comparative results with and without latent space regularization ?\"*\n\n\n**A:** Thanks for your valuable suggestion! As shown in the supplement pdf file, we visualize the feature space with (w LSCR) or without (w/o LSCR) latent space regularization by using T-SNE (t-distributed Stochastic Neighbor Embedding) [P1]. T-SNE is an unsupervised non-linear dimensionality reduction technique for data exploration and visualizing high-dimensional data. By our latent space regularization, for those pixels/points with the same semantics (the same colour), their feature space tends to be more compact. Meanwhile, for those pixels/points with different semantics (the different colours), their feature space tends to be more distinguishable.\n\n\n---\n> ***Limitations 1:** \"The method proposed assumes SAM to be perfect in the sense of providing meaning masks.\"*\n\n**A:** Good points! Indeed, the label refinement module heavily relies on the SAM's prediction, and SAM is not always perfect. We observe it may fail in bad weather conditions, where the scenes are burr and caliginous. However, the label refinement module is just one of the components of our Cross-modality Noisy Supervision method. The other two modules, Latent Space and Prediction Consistency Regularization, are able to further mitigate error propagation and prevent the networks from overfitting the noisy labels (lines 53-60). The Ablation experiments in Table 2 also verify the effectiveness of these modules. We have supplemented the discussion in the revised version. Thanks!\n\n---\n**References:**\n- [P1] L. Van der Maaten and G. Hinton. Visualizing High-Dimensional Data Using t-SNE. JMLR, 2008.\n"}]

Author rebuttal:
We sincerely appreciate all reviewers for their insightful comments and help to improve our paper. 

---
We are encouraged that the reviewers acknowledging this work:
- "*the paper is well-written and easy to follow* (Reviewer 9W6n, Reviewer Mxk9, Reviewer inkR)";
- "*the overall method is both simple and effective* (Reviewer Mxk9)";
- "*the performance is: a significant enhancement* (Reviewer rM41) / *good* (Reviewer inkR) / *impressively perfect* (Reviewer Mxk9)". 

---
We would like to re-emphasize the main contributions of this work:
- In this work, we study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data.
- We propose a novel Cross-modality Noisy Supervision framework to effectively and synchronously train 2D and 3D networks with severe label noise, including prediction consistency and latent space consistency regularization schemes.
- Experiments conducted on indoor and outdoor datasets show that our method significantly outperforms state-of-the-art methods on 2D and 3D semantic segmentation tasks.

---
We upload a PDF file that contains two supplementary figures:
- *The **improved main figure** to better demonstrate our framework*
- *The **T-SNE visualization of the feature space** with or without latent space regularization*

We address the concerns below and will incorporate all feedback. We will actively participate in the Author-Reviewer discussion session. Please don't hesitate to tell us if there is anything unclear. Thanks!

---
Last but not least, we thank the PCs, ACs, and all the reviewers again for the time and effort devoted to this review.

Meta-review: Reviewers favor the proposed approach, and the majority vote favors the positive side. Authors provide comprehensive feedback on the concerns or questions raised by reviewers during the rebuttal phase. The additional supplement helps to resolve questions satisfactorily. AC read reviews and comments. Primarily, there was a series of discussions between reviewer rM41 and the authors, and even rM41 agreed with the effectiveness of the proposed approach. As a result, AC recommends the paper's acceptance. It is strongly advised to properly apply the requested modification and clarification in the revision.
Decision: Accept (poster)