Title: Weakly Supervised 3D Open-vocabulary Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D

open-vocabulary segmentation

neural radiance field
Abstract: Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, we distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF), which effectively lifts 2D features into view-consistent 3D segmentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text-image pairs. Code is available at https://github.com/Kunhao-Liu/3D-OVS.

Weakly Supervised 3D Open-vocabulary Segmentation

Abstract

Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, we distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF), which effectively lifts 2D features into view-consistent 3D segmentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and textimage pairs. Code is available at https://github.com/Kunhao-Liu/3D-OVS .

1 Introduction

Semantic segmentation of 3D scenes holds significant research value due to its broad range of applications such as robot navigation [1], object localization [2], autonomous driving [3], 3D scene editing [4], augmented/virtual reality, etc. Given the super-rich semantics in 3D scenes, a crucial aspect of this task is achieving open-vocabulary segmentation that can handle regions and objects of various semantics including those with long-tail distributions. This is a grand challenge as it necessitates a comprehensive understanding of natural language and the corresponding objects in the 3D world.

The main challenge in open-vocabulary 3D scene segmentation is the lack of large-scale and diverse 3D segmentation datasets. Existing 3D segmentation datasets like ScanNet [5] primarily focus on restricted scenes with limited object classes, making them unsuitable for training open-vocabulary models. An alternative is to distill knowledge from pre-trained 2D open-vocabulary segmentation models to 3D representations as learned with NeRF [6] or point clouds, by fitting the feature maps or segmentation probability outputs from the 2D models [4, 7]. Though this approach circumvents the need for the 3D datasets, it inherits the limitations of the 2D models which are usually finetuned with close-vocabulary datasets of limited text labels [8, 9], thereby compromising the open-vocabulary property, especially for text labels with long-tail distributions [2, 3].

∗

Open-vocabulary

a GPU card with fans

a beige mug

a black Nike shoe

a wooden ukulele

background

DINO

CLIP

NeRF

Rendered Novel Views

Segmentation Maps

Multi-view Images

͙

3D Representation

We achieve precise and annotation-free 3D open-vocabulary segmentation by distilling knowledge from two pre-trained foundation models into NeRF in a weakly supervised manner, supervised only by the open-vocabulary text descriptions of the objects in a scene, as illustrated in Fig. 1. One foundation model is CLIP [10] which is trained with Internet-scale text-image pairs [11] capturing extensive open-vocabulary multimodal knowledge. The other is DINO [12, 13] which is trained with largescale unlabelled images capturing superb scene layout and object boundary information. However, CLIP yields image-level features which are not suitable for pixel-level semantic segmentation. Thus certain mechanisms should be designed to extract pixel-level CLIP features without fine-tuning. Additionally, the image patches' CLIP features may have ambiguities for segmentation, which need to be regularized for accurate open-vocabulary segmentation. At the other end, DINO produces feature maps instead of explicit segmentation maps. Certain distillation techniques should be designed to extract the necessary information from DINO features to facilitate precise segmentation.

We construct a hierarchical set of image patches to extract pixel-level features from image-level CLIP features and design a 3D Selection Volume to identify the appropriate hierarchical level for each 3D point, effectively aligning CLIP features with pixel-level features without fine-tuning. In addition, we introduce a Relevancy-Distribution Alignment (RDA) loss to address CLIP feature ambiguities, aligning segmentation probability distribution with class relevancies that capture similarities between class text features and corresponding CLIP features. Moreover, we propose a novel Feature-Distribution Alignment (FDA) loss to distill object boundary information from DINO features. The FDA loss encourages close segmentation probability distributions for points with similar DINO features and distant distributions for dissimilar features. To address the training instability due to diverse distribution shapes, we further re-balance weights associated with similar and dissimilar DINO features.

Our method enables weakly supervised open-vocabulary segmentation of 3D scenes with accurate object boundaries. By distilling knowledge from CLIP without fine-tuning, our approach preserves its open-vocabulary knowledge and effectively handles text labels with long-tail distributions. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Remarkably, our experiments demonstrate that our method surpasses fully supervised models trained with segmentation annotations in certain scenes, highlighting the possibility that 3D open-vocabulary segmentation can be effectively learned from large amounts of 2D images and text-image pairs.

In summary, the contributions of this work are three-fold. Firstly , we propose an innovative pipeline for weakly supervised 3D open-vocabulary segmentation by distilling knowledge from pre-trained foundation models into NeRF without requiring any annotations in training. Secondly , we introduce a Selection Volume to align image-level CLIP features with pixel-level features, supplemented by novel Relevancy-Distribution Alignment and Feature-Distribution Alignment losses that respectively resolve CLIP features' ambiguities and effectively distill DINO features for 3D scene segmentation. Lastly , extensive experiments demonstrate that our method effectively recognizes long-tail classes and produces accurate segmentation maps, even with limited input data.

2 Related Work

Open-vocabulary Segmentation. In recent years, the field of 2D open-vocabulary segmentation has garnered significant attention, driven by the availability of extensive text-image datasets and vast computational resources. Predominant approaches [8, 14-19] typically distill knowledge from large-scale pre-trained models, such as image-text contrastive learning models [10, 20-22] and diffusion models [23]. However, the distillation process requires fine-tuning on close-vocabulary datasets, contrasting with massive datasets used for large-scale pre-trained models [11]. This leads to limited performance in recalling infrequent classes with long-tail distributions [2, 3], compromising the open-vocabulary property. OpenSeg [24] is not finetuned on a closed set of classes but is weakly supervised via image captions. However, OpenSeg has a smaller vocabulary and knowledge than CLIP as it is trained on a much smaller dataset. Our method, without fine-tuning CLIP, effectively handles such classes.

3D Scenes Segmentation. 3D scene segmentation has been a long-standing challenge in computer vision. Traditional approaches focus on point clouds or voxels with limited class variety in datasets, restricting generalizability to unseen classes [5, 25-36]. Recently, numerous point-cloud-based techniques have emerged to explore open-vocabulary 3D scene segmentation by encoding 2D openvocabulary models' features into 3D scene points [3, 7, 37-40]. However, these methods are also mostly evaluated on datasets with restricted scenes and limited class ranges [5, 27, 28, 41], not fully exhibiting the open-vocabulary property. Moreover, point clouds have compromised geometric details, making them less suitable for precise segmentation compared to NeRF representations [6, 42]. Consequently, there has been a surge in NeRF-based 3D segmentation techniques that mainly address interactive segmentation [4, 43, 44], panoptic segmentation [45, 46], moving part segmentation [47], object part segmentation [48], object co-segmentation [49], unsupervised object segmentation [50, 51], etc. FFD [4] attempts to segment unseen text labels during training by fitting LSeg's [8] feature maps to a NeRF, but inherits LSeg's limitations, hindering generalization to long-tail distribution classes. Our method overcomes these challenges by directly using CLIP image features and distilling them into a NeRF representation [42] without fine-tuning on close-vocabulary datasets.

Foundation Models. Pre-trained foundation models [52, 53] have become a powerful paradigm in computer science due to their ability to capture general knowledge and adapt to various downstream tasks [10, 12, 13, 20, 23, 54-57]. These models are trained using various paradigms in natural language processing, such as masked language modeling [57, 58], denoising autoencoder [59], replaced token detection [60], and sentence prediction tasks[61], as well as in computer vision, including data generation [23, 62, 63], data reconstruction [64], and data contrastive learning [10, 12, 13, 20-22]. Foundation models acquire emergent capabilities for exceptional performance on downstream tasks, either in a zero-shot manner or with fine-tuning. In this work, we harness the capabilities of two prominent foundation models, CLIP [10] and DINO [12, 13]. CLIP learns associations between images and texts by mapping them to a shared space, facilitating applications in tasks like image classification, object detection, visual question-answering, and image generation [9, 10, 23, 62, 65, 66]. DINO, trained in a self-supervised manner, extracts scene layout information, particularly object boundaries, and has been successfully employed in tasks such as classification, detection, segmentation, keypoint estimation, depth estimation, and image editing [12, 13, 49, 67-69].

3 Method

We propose a novel method for weakly supervised open-vocabulary segmentation of reconstructed NeRF. Given the multi-view images of a scene and the open-vocabulary text description for each class, we aim to segment the reconstructed NeRF such that every 3D point is assigned a corresponding class label.

To achieve this, we exploit the CLIP model's multimodal knowledge by mapping each 3D point to a CLIP feature representing its semantic meaning. As CLIP only generates image-level features, we extract a hierarchy of CLIP features from image patches and learn a 3D Selection Volume for pixel-level feature extraction, as described in Sec. 3.1.

However, the image patches' CLIP features may have ambiguities for segmentation, showing inaccurate absolute relevancy values which lead to misclassification. Take an image patch capturing an apple lying on a lawn as an example. The corresponding CLIP feature contains both apple and lawn information. If the apple is relatively small, the patch could be classified as lawn because lawn dominates the patch's CLIP features. Then the class apple would be ignored. To address this issue, we introduce a Relevancy-Distribution Alignment (RDA) loss, aligning the segmentation probability distribution with each class's normalized relevancy map, as described in Sec. 3.2. For precise object boundaries, we align the segmentation probability distribution with the images' DINO features. Previous segmentation approaches utilizing DINO features have focused on unsupervised segmentation [49, 68], lacking semantic meaning for segmented parts. In the open-vocabulary context, assigning accurate text labels to segmented regions requires aligning DINO feature clusters with correct text labels. To overcome this challenge, we propose a Feature-Distribution Alignment (FDA) loss, segmenting the scene based on DINO features' distribution and assigning appropriate text labels, as described in Sec. 3.3.

3.1 Distilling Pixel-level CLIP Features with a 3D Selection Volume

We propose a method based on multi-scale and multi-spatial strategies to adapt CLIP's image-level features for pixel-level segmentation, motivated by the observation that a pixel's semantic meaning should remain invariant to its surrounding pixels. The multi-scale component extracts features from patches of varying sizes around each pixel, and the multi-spatial component extracts from patches in which each pixel is at different positions. We average the multi-spatial features in each scale, attributing the primary direction of these features to the pixel's semantic meaning. We utilize a sliding-window algorithm for multi-spatial feature extraction. To prevent checkerboard patterns in the features and potential segmentation artifacts, we introduce randomness in the window size. The algorithm's pseudo-code is in Alg. 1.

After extracting pixel-level features from CLIP, we now have the multi-view RGB images and their corresponding multi-scale pixel-level features. Each ray r is assigned its multi-scale feature F ( r ) ∈ R N s × D for supervision. Rather than simply averaging each ray's multi-scale features across scales, we introduce a 3D Selection Volume S to determine the most suitable scale indicative of the object size within a patch. Each 3D point x ∈ R 3 yields a selection vector S x ∈ R N s from S . Following [4, 45], we introduce an additional branch to render the CLIP feature. We can then render the RGB value, the CLIP feature, and the selection vector of each ray r using volume rendering [6]:

Relevancy

Normalize

class

label

Spatial Location

ഥ

ݎ

௔

ഥ

ݎ

௕

ܽ

ܾ

Region 1

Region 2

Spatial Location

Relevancy

ݎ

௔

ݎ

௕

ܽ

ܽ

Region 1

Region 2

class

label

where C i , F i , S i are the color, feature, and selection vector of each sampled point along the ray, T i = Π i -1 j =0 (1 -α i ) is the accumulated transmittance and α i = 1 -exp ( -δ i σ i ) is the opacity of the point. We apply a Softmax function to the selection vector of each ray such that the sum of the probability of each scale is equal to 1.

For a set of rays R in each training batch, the supervision loss can then be formulated as the combination of the L2 distance between rendered and ground truth RGB values and the cosine similarities cos ⟨ , ⟩ between the rendered features and the selected multi-scale CLIP features:

Given a set of text descriptions { [CLASS] i } C i =1 of C classes and the CLIP text encoder E t , we can get the classes' text features T = E t ( [CLASS] ) ∈ R C × D . Then we can get the segmentation logits z ( r ) of the ray r by computing the cosine similarities between the rendered CLIP feature and the classes' text features:

We can then get the class label of the ray l ( r ) = argmax ( z ( r )) .

3.2 Relevancy-Distribution Alignment for Ambiguity Mitigation

To mitigate the ambiguities of the CLIP features, we propose to align the segmentation probability distribution with the spatially normalized relevancy maps of each class, enabling our method to identify specific image regions described by each class text, as illustrated in Fig. 2. The segmentation probability of each ray P ( r ) can be derived from the segmentation logits with a Softmax function:

The relevancy of a given class is determined by the similarity between the class's text feature and the selected feature from the hierarchy of image patches' CLIP features. Given an image I , we can get its multi-scale pixel-level CLIP feature F I ∈ R N s × D × H × W using Alg. 1 and selection vector S I ∈ R N s × H × W using Eq. (2). And then we can get the image's relevancy map R I ∈ R C × H × W as:

where where h, w denotes the index in the H and W channel. We normalize each class's relevancy independently within an input view to [0 , 1] to mitigate the ambiguities of CLIP features, making our method discern image regions described by each class text:

Equally Differ

Equally Similar

Instable

Stable

Different

Similar

Class

Segmentation Probability

Target Distribution

where min () and max () are the functions getting the lowest and highest values across the spatial dimensions (i.e. H and W ). We apply a Softmax function to ¯ R I to make it a probability vector. Then we can assign each ray r its normalized relevancy with all the classes ¯ R ( r ) ∈ [0 , 1] C . We employ the Jensen-Shannon (JS) divergence to measure the discrepancy between the normalized relevancy ¯ R ( r ) and the segmentation probability distribution P ( r ) of each ray, formulating the Relevancy-Distribution Alignment (RDA) loss:

where M P ¯ R ( r ) = ( P ( r ) + ¯ R ( r )) / 2 is the average of the two distributions, and the subscript c denotes the probability of the c th class. By aligning the normalized relevancies and the segmentation probability distributions, our method can effectively identify the specific region corresponding to the text description of each class.

3.3 Feature-Distribution Alignment for Precise Object Boundary Segmentation

To ensure the segmentation exhibits precise object boundaries, we align the segmentation probability distribution with the images' DINO features, which have been shown to capture superb scene layouts and object boundary information [12, 13]. Following [49, 68], we extract the scene layout information with a DINO feature correlation tensor. Given a patch of size H p × W p , we can get the correlation tensor Corr _ F ∈ R H p W p × H p W p as:

whose entries represent the cosine similarity between the DINO features f at spatial positions ( h, w ) and ( i, j ) of the patch. In order to construct the correlation tensor for the segmentation probability distribution, we propose utilizing the JS divergence to assess the similarity between segmentation probabilities at two distinct spatial positions. The choice of JS divergence offers several advantages, including its symmetric nature and a bounded range of [0 , 1] , which contribute to improved numerical stability. However, since we only care about the class label of each point, i.e. the entry with the highest probability, we use a low temperature τ < 1 to get a sharper version of the segmentation probability distribution ´ P to let the model focus on the entry with the largest probability:

The distribution correlation tensor Corr _ D ∈ R H p W p × H p W p can thus be computed with:

where ´ P hwc , ´ P ijc are the segmentation probabilities of the c th class at spatial locations ( h, w ) and ( i, j ) of the patch, M ´ P ´ P = ( ´ P hw + ´ P ij ) / 2 is the average of the two distributions. Thus the correlation loss [68] can be expressed as:

where b is a hyper-parameter denoting that we consider the segmentation probabilities of two spatial locations ( h, w ) and ( i, j ) to be similar if their DINO features' similarity is larger than b and distant if less than b . Nonetheless, the correlation loss L corr introduces significant instability due to the diverse shapes of distributions with large divergence from a target distribution, making the loss assign wrong labels to the segmented parts. Conversely, when a distribution displays a low JS divergence with the target distribution, it consistently demonstrates a similar shape to the target distribution, as shown in Fig. 3. Based on this observation, we propose re-balancing the weights associated with similar and dissimilar DINO features. Specifically, we allocate a much greater weight to the correlation loss arising from similar DINO features and a smaller weight to that of dissimilar DINO features, thereby mitigating the instability caused by the correlation loss. Thus the Feature-Distribution Alignment (FDA) loss can be formulated with:

where clamp ( , min/max = 0) is to clamp all the elements smaller/greater than 0 to 0, thus making pos _ F ≥ 0 and neg _ F ≤ 0 , count_nonzero () is to count the number of non zero elements, and λ pos , λ neg are the weights associated with similar and dissimilar DINO features, which are set as λ pos = 200 ≫ λ neg = 0 . 2 by default.

The total training loss is: L = L supervision + L RDA + L FDA , where L supervision , L RDA are calculated with randomly sampled rays and L FDA is calculated with randomly sampled patches.

4 Experiments

Weevaluate our method on 3D open-vocabulary segmentation, showing that our method can recognize long-tail classes and produce highly accurate object boundaries even with limited input data. We employ TensoRF [42] as the backbone and extract 3 scales of pixel-level CLIP features. More implementation details and experiments are in the appendix.

Dataset. Existing 3D segmentation datasets predominantly focus on either restricted scenes with a narrow range of object classes [5, 70], or individual objects [71-73], thereby limiting their capacity to fully assess the task of 3D open-vocabulary segmentation. Thus following [2], we create a dataset comprising 10 distinct scenes. Each scene features a set of long-tail objects situated in various poses and backgrounds. Ground truth masks for the test views are manually annotated, enabling both qualitative and quantitative evaluation of our segmentation methods. We also evaluate our method on more diverse datasets which include human body, human head, indoor scenes with low-quality images [70, 74], and a complex scene from LERF datasets [2], as shown in the appendix.

Baselines. We benchmark our method with three NeRF-based methods capable of 3D openvocabulary segmentation: FFD [4], Semantic-NeRF (abbreviated as Sem) [45], and LERF [2].

View 1

View 2

baseball

dinosaur

rabbit

shrilling chicken

weaving basket

wood

View 1

View 2

New York Yankees cap

black headphone

green lawn

hand soap

red apple

stapler

View 1

View 2

Pikachu

Xbox wireless controller

a stack of UNO cards

grey sofa

FFD

Sem(OV-Seg)

LERF

Ours

Ground Truth

RGB

FFD, a pioneering effort in 3D open-vocabulary segmentation, applies the feature maps of the 2D open-vocabulary segmentation method, LSeg [8], to achieve its segmentation results. For Sem, we adapt it to the open-vocabulary segmentation context by distilling segmentation results from two state-of-the-art 2D open-vocabulary segmentation models: the diffusion-model-based ODISE [16], and the CLIP-based OV-Seg [19]. LERF closely aligns with our proposed approach due to its use of knowledge distillation from CLIP and DINO. However, its primary focus is on object localization rather than segmentation. We use the same scale level number and patch sizes in LERF for fair comparisons. We also include results obtained by independently segmenting each test view using the aforementioned 2D models. Note that under our settings, FFD, Sem are fully supervised methods using segmentation annotations.

4.1 Results

We present the qualitative results in Fig. 4 and quantitative results in Tab. 1. Our proposed method outperforms all other techniques, including those which heavily rely on extensive segmentation annotations, such as LSeg, ODISE, OV-Seg. In particular, ODISE and FFD underperform in our evaluation, as they are unable to identify many long-tail classes, suggesting that the fine-tuning process of LSeg and ODISE may have led to a significant loss of the open-vocabulary knowledge originally encapsulated by CLIP and Stable Diffusion [23]. OV-Seg attempts to retain CLIP's knowledge by leveraging a mined dataset, however, it requires a mask proposal model which produces inconsistent segmentation across views, making Sem(OV-Seg) produce noisy and imprecise segmentation. LERF

Ground Truth

w/o RDA

w/o FDA

Ablations

50% Views

10% Views

Single Scale

Single & 10%

Portuguese egg tart

dressing doll

green grape

mini offroad car

orange cat

pebbled concrete wall

wood

w/o re-balance

RGB

also fails to capture precise object boundaries due to its usage of a relatively naïve regularization loss, which fails to fully exploit the object boundary information within the DINO features. In contrast, our method exhibits robust performance, successfully recognizing long-tail classes and generating accurate and well-defined boundaries for each class. However, LERF allows querying any object without the need for running optimization again, which is an advantage over our method.

4.2 Studies

Ablations. We conduct ablation studies to evaluate the individual contributions of the RDA loss and the FDA loss to the overall performance of our proposed method. As shown in Tab. 2, both RDA loss and FDA loss are crucial to our method, without each of which can result in severe performance degradation. As illustrated in Fig. 5, without the RDA loss, the model does not resolve the ambiguities of the CLIP features, leading to misclassifications. For instance, it fails to distinguish between an orange cat and a Portuguese egg tart , and confuses a mini offroad car with wood . Without the FDA loss, although our method can correctly locate each class, it fails to segment precise object boundaries. When discarding the re-balancing in the FDA loss, i.e. using the correlation loss [68], the model produces accurate boundaries but assigns each cluster the wrong label due to the instability brought by diverse distribution shapes.

Limited input. Given the substantial computational and storage demands of extracting hierarchical CLIP features for each view (exceeding 1GB for 3 scales in the full model), we explore whether reducing input CLIP features would yield similar results, as shown in Tab. 2 and Fig. 5. We test two modifications: reducing views for feature extraction and using a single scale of CLIP features rather than three. Halving the input views for feature extraction leads to negligible performance degradation ( < 1% ) and minor visual differences compared to the full model. When reducing to only 10% of input views, equivalent to 2-3 views in our dataset, we observe a modest 9% drop in the mIoU score and a 1% decrease in the Accuracy score, while retaining accurate segmentation across most classes. Using a single scale of CLIP features also only incurs minimal degradation ( < 1% ). Even under extreme conditions, i.e., extracting a single scale of features from 10% of input views (total only 1 / 30 input of the full model), performance degradation is limited to 10% . This efficient approach even outperforms LERF [2] which utilizes all input views and scales, highlighting our method's robustness.

5 Limitations

The limitations of our method are twofold. First, unlike LERF [2], our method requires text labels before training. To perform segmentation with new text labels, our method needs to be retrained.

Inferring accurate boundaries with open vocabularies is challenging for implicit representations like NeRF, as NeRF learns a continuous representation rather than a discrete one. It is promising to learn object-level discrete representation using NeRF in future work.

Second, since our method has never seen any segmentation maps during training (only weakly supervised by the text labels), it fails to segment complex scenes like indoor datasets [70, 74] with high precision, as shown in the appendix. Our method distills pixel-level CLIP features in a patchbased fashion with a strong inductive bias for compact objects with an aspect ratio close to 1. For objects with large complex shapes and unobvious textures, our method would fail to recognize them.

6 Conclusion

In this study, we address the challenge of 3D open-vocabulary segmentation by distilling knowledge from the pre-trained foundation models CLIP and DINO into reconstructed NeRF in a weakly supervised manner. We distill the open-vocabulary multimodal knowledge from CLIP with a Selection Volume and a novel Relevancy-Distribution Alignment loss to mitigate the ambiguities of CLIP features. In addition, we introduce a novel Feature-Distribution Alignment loss to extract accurate object boundaries by leveraging the scene layout information within DINO features. Our method successfully recognizes long-tail classes and produces precise segmentation maps, even when supplied with limited input data, suggesting the possibility of learning 3D segmentation from 2D images and text-image pairs.

References

Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam. Clipfields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663 , 2022. 1

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553 , 2023. 1, 3, 7, 9, 16, 17

Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241 , 2023. 1, 3

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. arXiv preprint arXiv:2205.15585 , 2022. 1, 3, 4, 7, 17

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017. 1, 3, 7

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65 (1):99-106, 2021. 1, 3, 4

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211.15654 , 2022. 1, 3

Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546 , 2022. 1, 3, 7, 8

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1679316803, 2022. 1, 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from

natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 2, 3

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021. 2, 3

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9630-9640, 2021. 2, 3, 6

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023. 2, 3, 6

Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. arXiv preprint arXiv:2211.14813 , 2022. 3

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18134-18144, 2022.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Openvocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04803 , 2023. 7, 8

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270 , 2022.

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1158311592, 2022.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210.04150 , 2022. 3, 7, 8

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021. 3

Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 , 2021.

Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets languageimage pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 529-544. Springer, 2022. 3

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022. 3, 8

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 540-557. Springer, 2022. 3

Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105 , 2017. 3

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9297-9307, 2019.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020. 3

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 , 2017. 3

Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX , pages 202-221. Springer, 2020.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354-3361. IEEE, 2012.

Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Craig Yu, and Sai-Kit Yeung. Scenenn: A scene meshes dataset with annotations. 2016 Fourth International Conference on 3D Vision (3DV) , pages 92-101, 2016.

Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45: 3292-3310, 2021.

Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 909-918, 2019.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2446-2454, 2020.

Lyne P. Tchapmi, Christopher Bongsoo Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. 2017 International Conference on 3D Vision (3DV) , pages 537-547, 2017.

Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4558-4567, 2018. 3

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language-driven open-vocabulary 3d scene understanding. arXiv preprint arXiv:2211.16312 , 2022. 3

Kirill Mazur, Edgar Sucar, and Andrew J Davison. Feature-realistic neural fusion for real-time, open set scene understanding. arXiv preprint arXiv:2210.03043 , 2022.

Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d visionlanguage models. In Conference on Robot Learning , 2022.

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.04926 , 2023. 3

Iro Armeni, Ozan Sener, Amir Roshan Zamir, Helen Jiang, Ioannis K. Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1534-1543, 2016. 3

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision , 2022. 3, 7, 15

Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. arXiv preprint arXiv:2212.13545 , 2022. 3

Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. 2022 International Conference on 3D Vision (3DV) , pages 443-453, 2022. 3

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15818-15827, 2021. 3, 4, 7

Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. arXiv preprint arXiv:2212.09802 , 2022. 3

Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. 2021 International Conference on 3D Vision (3DV) , pages 910-919, 2021. 3

Jesus Zarzar, Sara Rojas, Silvio Giancola, and Bernard Ghanem. Segnerf: 3d part segmentation with neural radiance fields. arXiv preprint arXiv:2211.11215 , 2022. 3

Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776 , 2022. 3, 4, 6

Shengnan Liang, Yichen Liu, Shangzhe Wu, Yu-Wing Tai, and Chi-Keung Tang. Onerf: Unsupervised 3d object segmentation from multiple views. arXiv preprint arXiv:2211.12038 , 2022. 3

Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148 , 2021. 3

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021. 3

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419 , 2023. 3

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 2019. 3

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. 3

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics , 8:64-77, 2020. 3

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019. 3

Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 , 2020. 3

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. 3

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. 3

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022. 3

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022. 3

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11686-11695, 2022. 3

Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. In Annual Meeting of the Association for Computational Linguistics , 2022. 3

Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10738-10747, 2022. 3

Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414 , 2022. 4, 6, 9

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2(3):4, 2021. 3

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019. 7, 10, 17

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051 , 2022. 7

Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. arXiv preprint arXiv:2301.07525 , 2023.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10901-10911, 2021. 7

Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10912-10922, 2021. 7, 10, 17

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. 16

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) , 2019. 16

Appendix

This document provides supplementary materials for Weakly Supervised 3D Open-vocabulary Segmentation in implementation details (Appendix A), more ablations (Appendix B), more evaluations (Appendix C), and more results (Appendix D).

A Implementation Details

A.1 Model Architecture

Density

Density Volume

Selection Vector

Selection Volume

MLP

MLP

View Direction

RGB

CLIP Feature

Shared Volume

We use TensoRF [42] as our base NeRF architecture for efficiency, the plane size is also the same as the default setting of TensoRF. The RGB and CLIP feature branches share the same volume and use the same intermediate features. The selection volume and density volume are two other independent volumes. We directly use the features extracted from the selection volume and density volume as the selection vector and the density value, as they have low dimensions and are view-independent. We use the original MLP architecture in TensoRF to extract the view-dependent RGB value and use another MLP which discards view direction input to extract the rendered CLIP feature. The architecture is illustrated in Fig. 6.

A.2 Hyperparameters

We set τ = 0 . 2 to get the shaper segmentation probability distribution ´ P . The offset b is set to 0 . 7 to measure the similarities of the DINO features, meaning that two DINO features are considered similar if their cosine similarity is larger than 0 . 7 , and different if less than 0 . 7 . We use 3 scales of CLIP features, and the patch sizes of each scale are set as s/ 5 , s/ 7 , and s/ 10 , where s is the smaller value in the width and height of the input image I . In the ablation studies, we use s/ 7 as the patch size of the single-scale CLIP feature input. The weights associated with similar and dissimilar DINO features in L FDA are set as λ pos = 200 and λ neg = 0 . 2 by default. In certain scenes, we find that setting λ neg to 0.22 or 0.18 can produce better results. We use ViT-B/16 CLIP model to extract the image and text features and use version 1 dino_vitb8 model to extract the DINO features because it employs the smallest downsampling factor of 8 which is advantageous for high-precision segmentation.

A.3 Training

To reconstruct a NeRF from multiview images of a scene, we follow the same training settings as TensoRF. For segmentation training, we train the model for 15k iterations. In the first 5k iterations,

we freeze the shared volume and density volume, and train the selection volume and the CLIP feature branch. For the rest 10k iterations, we further finetune the shared volume and the RGB branch. We use Adam optimizer with betas = (0 . 9 , 0 . 99) . The learning rates for training the volume and MLP branch are respectively set to 0 . 02 and 1 e -4 . For finetuning the volume and the MLP, the learning rates are set to 5 e -3 and 5 e -5 . We also employ a learning rate decay with a factor of 0 . 1 .

The multi-scale pixel-level CLIP features of training views are pre-computed before training and the DINO features are computed with sampled patches on the fly during training. When computing L supervision and L RDA , we randomly sample rays with a batch size of 4096. When computing L FDA we randomly sample patches of size 256 × 256 with a batch size of 8. We use a downsampling factor of 8 when sampling rays and a factor of 5 when sampling patches. The model is trained on an NVIDIA A5000 GPU with 24G memory for ∼ 1h30min for each scene.

A.4 Dataset

We capture 10 scenes using smartphones and use Colmap [75] to extract camera parameters for each image. We capture 20 ∼ 30 images for each scene and the resolution of each image is 4032 × 3024 . We follow the data structure of LLFF [76]. We manually annotate the segmentation maps of 5 views for each scene as the ground truth for evaluation.

We list the text labels used in our experiments in Tab. 3. Note that we sometimes choose to add a general word to describe the whole background, such as wall , desktop , following LERF [2]. The text labels in our dataset contain many long-tail classes, which can be used to fully assess the open-vocabulary capability.

Ground Truth

Portuguese egg tart

dressing doll

green grape

mini offroad car

orange cat

pebbled concrete wall

wood

RGB

w/o

!"#_%

w/o Selection

View

1

View 2

T-shirt

arms

head

jeans

white wall

RGB

LERF

ours

RGB

LERF

ours

black T-shirt

eyes

hair

mouth

neck

face

white wall

nose

B More Ablations

We perform two more ablation studies on the Selection Volume and the FDA loss, as shown in Tab. 4 and Fig. 7. Without the Selection Volume, we simply average the multi-scale CLIP features rather than learning the appropriate scale. We can see that both the mIoU score and the Accuracy score are inferior to the full model. We could discard the dissimilar part neg _ F since dissimilar DINO features often impair the stability of the correlation loss. However, neg _ F encourages different segmentation probabilities for different semantic regions and it plays a crucial role for precise object boundary extraction.

C More Evaluations

We additionally perform evaluations on human body, human head, indoor datasets with low-quality images [70, 74], and a complex scene from LERF datasets [2]. We compare with the concurrent work LERF qualitatively due to the lack of labels or the defective annotations as pointed out in [4]. We also perform experiments with different text prompts. We use the same scale level number and patch sizes in all comparisons.

Human body and head. As shown in Fig. 8, our method segments more precise parts than LERF. Specifically, LERF fails to segment the "head" in the human body and the "black T-shirt" in the human head. In contrast, our method can recognize and segment these parts correctly because our designed RDA loss addresses the ambiguity of the CLIP features effectively.

Indoor scenes with low-quality images. Fig. 9 shows experiments on the indoor datasets [70, 74], where many images are unrealistically rendered with less photorealistic appearances (as indicated in [4]) and have limited spatial resolution ( 640 × 480 or 1024 × 768 ). Due to these data constraints,

View

1

View 2

grey sofa

old television

white wall

window

RGB

LERF

ours

RGB

LERF

ours

coffee table

black and white photo

window blinds

lime wall

white sofa with cushion

wool carpet

View

1

View 2

a glass of red tea

apple

bear with brown fur

coffee mug

concrete ground

RGB

LERF

ours

View1

View2

View3

blue partition

desktop

red apple

RGB

ours

paper napkin

plate with cookies

sheep

wood table

our method sometimes confuses labels with similar appearances. However, we can see that our method still outperforms LERF by successfully segmenting more labels.

Complex scenes. Fig. 10 (left) shows the segmentation of one challenging sample from the LERF dataset, where the scene has complex geometry as well as many objects of varying sizes. It can be observed that LERF cannot segment most objects due to the ambiguities of CLIP features while our method can segment more objects correctly with more precise boundaries. Fig. 10 (right) shows a scene with multiple instances of a same class. Since instances of the same class often share similar appearance, texture, etc., they also have similar DINO features. As a result, FDA will not mistakenly segment them into different classes. The RDA loss will further help by assigning all these instances to the same text label. In the experiment, we observed that our method successfully segments all three apples into the same class with accurate boundaries.

View

1

View 2

Portuguese egg tart  ->  Pastel de Nata

dressing doll  ->  barbie

RGB

ours-original

ours-rephrased

black headphone  ->  listen to music

hand soap  ->  wash hand

RGB

ours-original

ours-rephrased

Segmentation with different text prompts. We conduct experiments to segment scenes with different text prompts. In the experiments, we replaced the original texts with different languages (e.g., Portuguese egg tart -> Pastel de Nata), names (e.g., dressing doll -> Barbie), and actions (e.g., hand soap -> wash hand, black headphone -> listen to music). As Fig. 11 shows, with the rephrased text prompts, our method can still segment the scenes reliably. The experiments are well aligned with the quantitative experiments as shown in Tab. 5.

D More Results

We show more segmentation visualizations of our method in Fig. 12 (bed), Fig. 13 (sofa), Fig. 14 (lawn), Fig. 15 (room), Fig. 16 (bench), Fig. 17 (table), Fig. 18 (office desk), Fig. 19 (blue sofa), Fig. 20 (snacks), and Fig. 21 (covered desk). The quantitative results are listed in Tab. 6.

Ours

Ground Truth

banana

black leather shoe

camera

hand

red bag

white sheet

RGB

Ours

Ground Truth

Pikachu

Xbox wireless controller

a stack of UNO cards

grey sofa

RGB

Ours

Ground Truth

New York Yankees cap

black headphone

green lawn

hand soap

red apple

stapler

RGB

"K

K

Ours

Ground Truth

baseball

dinosaur

rabbit

shrilling chicken

weaving basket

wood wall

RGB

Ours

Ground Truth

RGB

Portuguese egg tart

dressing doll

green grape

mini offroad car

orange cat

pebbled concrete wall

wood

Ours

Ground Truth

a GPU card with fans

a beige mug

a black nike shoe

a wooden ukulele

lime wall

RGB

Ours

Ground Truth

a can of red bull drink

a pack of pocket tissues

a white keyboard

blue partition

desktop

the book of The Unbearable Lightness of Being

RGB

Ours

Ground Truth

a JBL bluetooth speaker

a bottle of perfume

a squirrel pig doll

an aircon controller

blue-grey sofa

sunglasses

RGB

BI

Ours

Ground Truth

Coke Cola

Glico Pocky chocolate biscuits sticks box

calculator

desktop

orange juice drink

pitaya

RGB

Ours

Ground Truth

Dove body wash

canned chili sauce

desktop

electric shaver

gerbera

RGB

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach for fusing in 3D the output of 2D neural networks (CLIP an DINO) applied to the analysis of multiple views of a given static scene. The main application is open-vocabulary 3D segmentation of radiance fields. Compare to trivially fusing CLIP features, the method introduces a few improvements, such as re-normalising the response of CLIP and using DINO to regularise the segments. These improvements are shown to help segmentation significantly. Comparison to recent works, including concurrent work such as LERF, are included.\n\nSTRENGTHS:\n* Fusing open-ended features such as CLIP in 3D can be useful in applications, including, as demonstrated here, for open-vocabulary 3D segmentation.\n\n* I appreciated the effort to compare to very recent *concurrent* work like LERF. The fact that the method outperforms the latter, at least in the authors' experiments, is a further bonus.\n\n* There are a few ideas (class re-balancing, multi-scale feature selection, relevance renormalisation and consistency with DINO features) which are shown to improve significantly the performance compared to simply fusing CLIP features. This illustrates some difficulty with the latter approach, and also shows simple fixes to these issues.\n\nWEAKNESSES:\n* The idea of fusing 2D features in 3D reconstruction using neural fields is not particularly novel as such. Semantic NeRF has done it for segmentation, N3F and FFD for unsupervised features like DINO, Panoptic NeRF and others for semantic segmentation, etc. FFD already included some early experiment with open-ended language-based features (and this paper compares to them).\n\n* The math needs significant revision. Some issues amounts to typos and inaccuracies, but others really make it hard to understand how the algorithm works in detail. Some aspects of the algorithm are also slightly surprising and need some discussion. See below:\n   \n     * Eq. (3) is the main loss that allows to learn the radiance field model, including its extension to predict the multi-scale CLIP features F and the scale selection vectors S. I understand most of this *except* why and how this can learn feature selection. First, the equation does not make literal sense. The inner product $\\langle \\hat F(\\mathbf{r}) \\cdot S(\\mathbf{r}) F(\\mathbf{r} \\rangle$ misses a comma, but I expect that the intended read is  $\\langle \\hat F(\\mathbf{r}) \\cdot S(\\mathbf{r}), ~ F(\\mathbf{r} \\rangle$. Second, the $\\cdot$ product is unclear. $F$ is a $N_s \\times D$ matrix and $S$ is a $N_s$-dimensional vector. Is this an element-wise product with broadcast? Thirdly, $\\hat F$ are the multi-scale CLIP features \"fused\" (i.e., rendered) by the radiance field and $F$ the ones predicted from the 2D image. Why comparing these two weighed by $S$ should result in $S$ learning to select the appropriate scale of CLIP features to use for that point?\n\n    * Eq. (6): Is this meant to be $\\langle T, F_I \\rangle$? The notation $\\langle T \\cdot F_I\\rangle$ does not make literal sense. Furthermore, $T$ is a $C\\times D$ matrix and $F_I$ is a $N_s\\times D\\times H\\times W$ matrix. The output is $C\\times H \\times W$, so we can sort of guess what the authors mean to do, but the notation does not show how the scale dimension $N_s$ is reduced (presumably using the scale selector $S$, but the notation does not reflect that).\n\n    * The RDA loss of Eq. (8) is a bit problematic. First, from a modelling perspective, it seems to be based on spatially re-normalizing the response of the CLIP features to all possible classes $c$ an a class-by-class basis (see also Fig. 2). This is done to prevent certain classes from \"swamping\" others. However, what happens if a class is *not* present at all in a scene? Wouldn't this *force* the class to still be segmented, thus incorrectly? Second, there are issues with Eq. (8) as such. The quantity $\\bar R(\\mathbf{r})$ is a $C$-dimensional non-negative vector, but, unless this is done implicitly, it does *not* sum to one. I.e., its is not a *probability* vector. Can you really interrupted Eq. (8) as a sum of JS divergences?\n\n    * Eq. (9)  what is the notation $\\cos f_{hw} \\cdot f_{ij}$? Do you mean $\\langle \\frac{f_{hw}}{\\|f_{hw}\\|},\\frac{f_{ij}}{\\|f_{ij}\\|} \\rangle$  ?\n\n    * What is $z$ in Eq. (9)?\n\n    * I understand that the authors are trying to implement the segmentation distillation idea of [49] -- Eq. (12) certainly bears resemblance to that work. However, I had a really hard time understanding the notation used in Section 3.3 overall.\n\n* The evaluation is only carried out on 10 custom scenes with comparatively little clutter. Given that the authors have been willing to compare to LERF (which is concurrent work, so this is definitely not required) I wonder if it would make sense to run the assessment on *their* 13 custom scenes as well.\n\nQUESTIONS:\n* See the technical issues above. Can you clarify the notation, and, especially, the question about renormalisation in the RDA loss?\n\n* In the scaling experiment of Table 2, bottom 4 rows, you use a very small number of views (according to the text, just 2-3 -- se line 279) are sufficient to obtain good segmentation performance. Do you mean that you are still using all *RGB* views for 3D reconstruction and just 2-3 views with CLIP features, or that you are using only 2-3 features in total, including for 3D reconstruction? The latter would be very surprising.\n\nLIMITATIONS:\nThere is no substantive discussion of limitations (e.g., no failure cases demonstrated, no section or paragraph discussing limitations). Note that the NeurIPS guidelines encourage explicitly the discussion of limitations.\n\nThere is no discussion of societal impact, ethics, copyright, data protection etc. I don't believe that any of this is particularly relevant to this paper though. I do not see direct potential harm stemming from this research, and the datasets used appear custom-collected and do not contain personal data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

comments: SUMMARY:
This paper introduces a method that incorporates pre-trained CLIP features into Nerf for 3D open-vocabulary segmentation. The authors also utilize pre-trained DINO features to regularize the segmentation process and achieve precise segmentation boundaries. Additionally, they propose a Relevancy-Distribution Alignment loss to address ambiguities in CLIP features.

STRENGTHS:
1. The paper demonstrates a fluent and well-structured organization.
2. The experimental results exhibit favorable performance.

WEAKNESSES:
1. The testing data is relatively simple, consisting of views with limited perspectives, and does not include results on larger and more complete 3D scenes (e.g., the benchmark proposed by Lerf).
2. Regarding the Feature-Distribution Alignment issue: If there are multiple objects of the same class, such as x1, x2, x3, in an image, their DINO features' similarity may be lower than b, but the Feature-Distribution Alignment could make them assigned to different classes, which is unreasonable. Although λ_neg is small, the supplementary material demonstrates the criticality of neg_F, and removing it significantly deteriorates the results.

QUESTIONS:
1. How would the performance be affected if the Relevancy-Distribution Alignment was not used during training and instead the segmentation probabilities were spatially normalized during inference?
2. How does the proposed method perform on Lerf's benchmark?

LIMITATIONS:
The processing of multi-spatial features of CLIP incurs substantial computational and storage resource requirements.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of 3D scene segmentation using CLIP and DINO distilled into a NeRF representation. Since CLIP and DINO are pre-trained, the method works in an unsupervised fashion i.e., no additional per-point training labels are needed. This is achieved by applying CLIP to small crops in a sliding window fashion resulting in pixel-aligned features. This is repeated for multiple scales, resulting in multiple pixel-aligned CLIP features maps, one for each scale. The paper introduces two new losses (FDA and RDA) that rely on a closed-set of classes during training of the NeRF. The method is evaluated on their own dataset and annotations, and compares to multiple, recent methods (LSeg, ODISE, OVSeg, LERF, FFD).\n\nSTRENGTHS:\nA novel “volume-selection” scheme used to align per-image CLIP features to per-pixel CLIP features, in addition to two new losses: Relevancy Distribution Alignment loss and Feature Distribution Alignment loss that do not require a dedicated pixel-aligned CLIP method.\n\nWEAKNESSES:\nI struggle a bit with the intermixing of the term “open-vocabulary” and the fact that the method relies on a known / closed set of classes C. E.g., “open-vocabulary text description for each class” (l.119) somehow implies that the set of classes that we want to segment is known a priori, whereas open-vocabulary implies that there are no such restrictions.\nImportantly, the two losses presented in Sec.3.2 and Sec.3.3 seem to rely on a predefined set of classes C - i.e. **each NeRF is actually optimized for a closed-set of classes, and if the classes change, the NeRF needs to be retrained**. The method can be seen as open-world in the sense that there is no inherent limitation to the classes, however an optimized NeRF is a closed-set representation. Is that correct? I think this aspect should be discussed in more detail in the limitations.\n\nThe writing could be improved, numerous aspects are still unclear and should be properly introduced and explained (Please see the minor points under Questions - I consider them minor since it can be improved in an updated version, however there are so many points that are unclear that the overall presentation of the method is limitted). \n\nThis paper follows an unfortunate trend that is observable across the field of LLM-based methods for open-set scene understanding: each paper presents a new method and at the same time introduces a new (relatively small) evaluation dataset on which they improve over prior/concurrent work. It is clear that such evaluations are not entirely convincing since the results could be cherry picked and favoring the newly proposed model. This paper is unfortunately no exception to that trend. In particular, the dataset selection is motivated by the fact that ScanNet [5] is too limited in terms of classes - however there was a recent extension to ScanNet200 that should mitigate this problem. Also Replica [70] does feature 50-80 semantic classes (depending on the scenes) which shows a long-tail distribution. Hence, there are existing datasets (maybe not perfect) but at least established ones and certainly not worse than very few newly recorded scenes. Further, since the proposed method compares with LERF, it could also compare on the LERF dataset.\n\nThe argumentation / motivation in l.28-31 is unfortunate: it argues that the major challenge of open-set segmentation is the lack of datasets, however that is not really the case since open-set datasets cannot exist as it would require an infinite amount of all potentially existing labels.\n\nThe contributions (1) (in line 71-73) - how does it differ from Decomposing Nerf [4] ? They also use CLIP and DINO in a very similar fashion. Instead of LSeg one could use the proposed multi-scale CLIP feature maps and then compare, this would be a reasonable baseline.\n\nQUESTIONS:\nHow is the selection vector S_x obtained? Eq. (2) shows that we can render it from the optimized NeRF, that is clear. However, we also need it already during the optimization, see Eq. (3). How does that work, i.e., how can the value S(r) that is only obtained via rendering of the optimized NeRF at the same time be used to optimize the NeRF?\nFinally, I do not see an ablation study that shows the benefits of the 3D Selection Volume. Does the 3D selection volume improve over simply averaging multi-scale features as described in l.151?\n\nMinor:\n\n- l.4 “lack of [...] open-vocabulary segmentation datasets” I would even argue that open-vocabulary datasets are inherently impossible as it would require exhaustive labeling of all possible queries.\n\n- l.7 ”finetuned with close-vocabulary datasets” this is only the case for a subset of visual-language models such as LSeg, other approaches are not fine tuned on closed sets of data e.g. OpenSeg.\n\n- l.35 same point as before: OpenSeg is not fine tuned on a closed set of classes but weakly supervised via image captions.\n\n- l.46 - ‘ambiguous similarities with text description’ what does that mean?\n\n- l.53-55 I don’t understand this sentence, can you break it down for me? What are CLIP feature ambiguities? What is a segmentation probability distribution? What are class relevancies? Until l.62 is similarly unclear - I think it’s mostly unclear because the used terminology is not properly introduced / defined. I can guess what is trying to be achieved but that is not my job, it should be clearly written.\n\n- l.126 “ambiguous similarities” - what does that mean?\n\n- l.127 what are “relevancy values”? Fig.2 is not that helpful since it also does not explain the terms. Again, what is a segmentation probability distribution (l.130)? I can sort of guess it (maybe I’m wrong, maybe I’m right?), why not clearly define and explain these terms since a large part of the main contribution depends on them?\n\n- l.169 Still unclear meaning of “ambiguities of the CLIP features”\n\nLIMITATIONS:\nLimitations are not addressed. For example, one could discuss the impact of the closed-set of classes C required to optimize the NeRF since this has implications on the practical application (see under weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

comments: SUMMARY:
This work addresses the open vocabulary segmentation by leveraging pretrained foundation model, DINO and CLIP. They firstly lifts 2D CLIP features into NeRF. Then, relevancy distribution alignment loss is used to mitigate the CLIP features's ambiguity. Feature distribution alignment loss is used to improve object boundaries from DINO feature.

STRENGTHS:
3D open vocabulary segmentation is an interesting and impactful topic. It is very challenging to use a fully supervised way for training such a model since annotation costs large amount of human labor. This paper leverages 2D foundation models. The presentation is fairly good and the proposed relevancy distribution alignment and feature distribution alignment losses are well motivated.

WEAKNESSES:
There are many unclear parts that make me feel hesitated to suggest acceptance of this paper.

1. how they used CLIP is not well justified. Will the patch-wise method break the CLIP feature space? I assume many patches covers only background or small portion of objects.

2. Why DINO? CLIP and DINO are two most important models for this work. So, motivating the utilization of them is important. Especially for DINO, why it is good for object boundaries. Visualization would help or ablation study will help.

3. the most concerning part is the experiments. The method is validated in limited settings. The class set is small that is not enough to show "open vocabulary" capability. The dataset is mall.

QUESTIONS:
See weaknesses.

LIMITATIONS:
The unclear parts in the paper and the limited experiments jointly make the performance of this work not convincing enough.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper trained a NeRF model leveraging on two self-supervised trained models CLIP and DINO. By distillating multi-view image features from these two large models, the NeRF model is able to synthesize accurate object segmentation masks on unseen scenes with long-tailed distributions. Experiments on 10 self-collected scenes shows that high-quality segmentation masks can be predicted after per-scene training without explicit segmentation annotations.\n\nSTRENGTHS:\n- This paper proposes a easy-to-follow pipeline to reach accurate open-world 3D segmentation by taking advantages of two pretrained large models CLIP and DINO without access to segmentation ground truth masks.\n\n- The experiments on self-collected scenes prove the capability of proposed method, and the ablations cover most of key design choices made in the paper, with adequate implementation details.\n\nWEAKNESSES:\n- Limited experiments on popular benchmarks.\n\n    My main concern lies on the limited evaluation as 'open world 3d segmentation' is a challenging topic. \n    - In this paper, 10 scenes with multi-view high-res images are taken for experiments. However, more diverse results on different datasets are expected to really prove the capability of ‘open-world 3D segmentation’. Existing datasets like ScanNet, VG150 still contains imbalanced long-tail distributions. How the proposed methods work on ScanNet, for example, where captured images are in VGA resolution and lower quality compared to used data in the paper.\n   - There are many hyper-parameters need to be considered during training. Are they sensitive to difference scenes/datasets?\n\n    Otherwise, the overall experiments may look too curated to prove the main statements.\n\n\n - One question to ask is whether the model sacrifice the ‘close-world’ capability to complement the open-world one. Therefore, it would be good to see how trained methods work on different categories with different types of text prompts.\n\n\n- What is the impact of text descriptions? Are the given text carefully chosen or some general text input covering key information are good enough for generating promising labels?\n\n\nOther comments:\n- Why only 6 of 10 scenes are shown for numerical experiments with baseline methods and 4 scenes are used for ablations? I see a complete 10 scene results of proposed methods in the supplement but not of baselines.\n- One question rather than weakness. Could authors explain why to choose to learn CLIP features with a shared TensorRF volume with colour instead of separately. Is there any specific reason or ablations for such design choice?\n\nQUESTIONS:\nPlease see weaknesses above where my main concern is the lack of analysis on diverse popular real-world scenes to really prove the model's capability.\n\nOverall this paper is attempting to tackle a challenging task and I think more discussion could better  strengthen the contributions.\n\nLIMITATIONS:\nLimitations and potential negative societal impacts have been discussed in the supplement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}