Weakly Supervised 3D Open-vocabulary Segmentation
Authors: Kunhao Liu
Fangneng Zhan
Jiahui Zhang
MUYU XU
Yingchen Yu
Abdulmotaleb El Saddik
Christian Theobalt
Eric Xing
Shijian Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D
open-vocabulary segmentation
neural radiance field
TL;DR: We address the challenge of 3D open-vocabulary segmentation by distilling knowledge from the pre-trained foundation models CLIP and DINO into reconstructed NeRF in a weakly supervised manner.

Abstract: Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, we distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF), which effectively lifts 2D features into view-consistent 3D segmentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text-image pairs. Code is available at https://github.com/Kunhao-Liu/3D-OVS.

Weakly Supervised 3D Open-vocabulary Segmentation
Kunhao Liu'
Fangneng Zhan
Jiahui Zhang'
Muyu Xu' Yingchen Yu' Shijian Lu
Abdulmotaleb El Saddik
Christian Theobalt
Eric Xing
Nanyang Technological University Max Planck
Institute for Informatics
University of Ottawa
*Carnegie Mellon University MBBZUA
Abstract
Open-vocabulary segmentation 3D scenes is fundamental function of human perception and thus crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open- open-voccabllayy segmentation models helps but it compromises the open- opca-vocabulary feature the 2D models are mostly finetuned with vocabulaayy datasets. We tackle the challenges in 3D pocc-uocyyylary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radi- ance field (NeRF), which effectively lifts 2D features into view-consistent 3D seg- mentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text- image pairs. Code is available at https: / / /ggtttb...b.... com/ cww/Kunhao-aa..aaaa Muunao-LLuu3D-O00
Introduction
Semantic segmentation of 3D scenes holds significant research value due to its broad range of applications such as robot navigation [1], object localization [2], autonomous driving [3], 3D scene editing [4], augmented/virtual reality, etc. Given the super-rich semanties in 3D scenes, crucial aspect of this task is achieving open-vocabulary segmentation that can handle regions and objects of various semanties including those with long-tail distributions. This is grand challenge as it necessitates a comprehensive understanding natural language and the corresponding objects the 3D world.
The main challenge open-vocabulary 3D scene segmentation is the lack of large-scale and diverse 3D segmentation datasets. Existing 3D segmentation datasets like ScanNet [5] primarily focus restricted scenes with limited object classes, making them unsuitable for training open-vocabulary models. An alternative is to distill knowledge from pre-trained 2D open-vocabulary segmentation models to 3D representations as learned with NeRF [6] or point clouds, by fitting the feature maps or segmentation probability outputs from the 2D models [4, 7]. Though this approach circumvents the need for the 3D datasets, inherits the limitations of the 2D models which are usually finetuned with close-vocabulary datasets of limited text labels [8, 9]. thereby compromising the open- apen-vocabulayy property, especially for text labels with long-tail distributions [2, 3].
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

3D Representation
Open-vocabulory Text Descriptions
Multi-view Imoges
wooden ukuiele
Rendered Noval
bogge mMA
Hatsune Mill statu0
bleck show
bockground
DINO
NeRF
CUP
Segmenatt on Maps
Figure 1: Weakly Supervised 3D Open-vocabulary Segmentation. Given the multi-view images of a 3D seene and the open- open-vocabulary text descriptions, our method distills open- open-vocabulary multimodal knowledge from CLIP and object reasoning ability from DINO into the reconstructed NeRF, producing accurate object boundaries for the 3D scene without requiring any segmentation annotations during training.
We achieve precise and annotation-free 3D open-vocabulary segmentation by distilling knowledge from two pre-trained foundation models into NeRF in a weakly supervised manner, supervised only by the open-vocabulary text descriptions of the objects in a scene, as illustrated in Fig. 1. One foundation model CLIP [10] which is trained with Internet-scale text-image pairs [11] capturing extensive open- vocabullyy multimodal knowledge. The other is DINO [12, 13] which is trained with large- scale unlabelled images capturing superb scene layout and object boundary information. However, CLIP yields image-level features which are not suitable for pixel-level semantic segmentation. Thus certain mechanisms should be designed to extract pixel-level CLIP features without fine-tuning. Additionally, the image patches' CLIP features may have ambiguities for segmentation, which need to be regularized for accurate open-vocabulary segmentation. At the other end, DINO produces feature maps instead of explicit segmentation maps. Certain distillation techniques should be designed to extract the necessary information from DINO features to facilitate precise segmentation.
We construct hierarchical set of image patches to extract pixel-level features from image-level CLIP features and design 3D Selection Volume to identify the appropriate hierarchical level for each 3D point, effectively aligning CLIP features with pixel-level features without fine-tuning. In addition, we introduce Relevancy-Distreee Distribution Alignment (RDA) loss to address CLIP feature ambiguities, aligning segmentation probability distribution with class relevancies that capture similar- ities between class text features and corresponding CLIP features. Moreover, we propose a novel Feuture- Diittir--Dittiittionnn Alignment (FDA) loss to distill object boundary information from DINO features. The FDA loss encourages close segmentation probability distributions for points with similar DINO features and distant distributions for dissimilar features. To address the training instability due to diverse distribution shapes, we further re- re-balance weights associated with similar and dissimilar DINO features.
Our method enables weakly supervised open- vocabulary segmentation of 3D scenes with accurate object boundaries. By distilling knowledge from CLIP without fine-tuning, our approach preserves its open- vocabuocayyl knowledge and effectively handles text labels with long-tail distributions. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Remarkably, our experiments demonstrate that our method surpasses fully supervised models trained with segmentation annotations in certain scenes, highlighting the possibility that 3D apccbuoocyyllayy segmentation can be effectively learned from large amounts of 2D images and text-image pairs.
In summary, the contributions of this work are threc-fold. for weakly supervised 3D open-vocabulary segmentation by distilling knowledge from pre-trained foundation models into NeRF without requiring any annotations in training. Secondly, we introduce a Selection Volume to align image-level CLIP features with pixel-level features, supplemented by novel Relevancy-Distre Distribution Alignment and Feature-Distributin Alignment losses that respectively resolve CLIP features' ambiguities and effectively distill DINO features for 3D scene segmentation. Lastly, extensive experiments demonstrate that our method effectively recognizes long-tail classes and produces accurate segmentation maps, even with limited input data.
Firstly, we propose an innovative pipeline

---

2 Related Work
Open-vocabulary Segmentation. In recent years, the field of 2D voca-vocyb segmentation has garnered significant attention, driven by the availability of extensive text-image datasets and vast computational resources. Predominant approaches [8, 14-19] typically distill knowledge from large-scale pre-trained models, such as image-text contrastive learning models [10, 20-22] and diffusion models [23]. However, the distillation process requires fine-tuning on close- vooab-vocay datasets, contrasting with massive datasets used for large-scale pre-trained models [11]. This leads to
limited performance in recalling infrequent classes with long-tail distributions [2, 3], compromising the open- vocabulary property. OpenSeg [24] not finetuned on closed set of classes but weakly supervised via image captions. However, OpenSeg has smaller vocabulary and knowledge than CLIP as it is trained on a much smaller dataset. Our method, without fine-tuning CLIP, effectively handles such classes.
3D Scenes Segmentation. 3D scene segmentation has been long-standing challenge in computer vision. Traditional approaches focus on point clouds or voxels with limited class variety in datasets, restricting generalizability to unseen classes [5, 25-36J. Recently, numerous conttcloud-based techniques have emerged to explore open-vocabulary 3D scene segmentation by encoding 2D open- vocabulary models' features into 3D scene points [3. 7, 37-40J. However, these methods are also mostly evaluated on datasets with restricted scenes and limited class ranges [5, 27, 28, 41]. not fully exhibiting the open- vocabulary property. Moreover, point clouds have compromised geometric details, making them less suitable for precise segmentation compared to NeRF representations [6, 42]. Consequently, there has been surge in NeRF-based 31) segmentation techniques that mainly address interactive segmentation [4, 43, 44], panoptic segmentation [45, 46], moving part segmentation [47], object part segmentation [48], object co-segmentation [49], unsupervised I [50,51]
etc. FFD [4] attempts to segment unseen text labels during training by fitting LSeg's [8] feature maps to NeRF, but inherits LSeg's 's limitations, hindering generalization long-tail distribution classes. Our method overcomes these challenges by directly using CLIP image features and distilling them into a NeRF representation [42] without fine-tuning on close-vocabulary datasets.
Foundation Models. Pre-trained foundation models [52, 53] have become powerful paradigm in computer science due to their ability to capture general knowledge and adapt to various downstream tasks [10, 12, 13, 20, 23, 54-57J. These models trained using various paradigms in natural language processing, such as masked language modeling [57, 58]. denoising autoencoder [59], replaced token detection [60], and sentence prediction [asks[61], as well as in computer vision, including data generation [23, 62, 63]. data reconstruction [64], and data contrastive learning [10, 12, 13, 20-22]. Foundation models acquire emergent capabilities for exceptional performance on downstream tasks, either in zero-shot manner or with fine-tuning. In this work, we harness the capabilities of two prominent foundation models, CLIP [10] and DINO [12, 13]. CLIP learns associations between images and texts by mapping them to shared space, facilitating applications in tasks like image classification, object detection, visual 10, 23, 62, 65, 66]. DINO, trained in self-supervised manner, extracts scene layout information, particularly object boundaries, and has been successfully employed in tasks such as classification, detection, segmentation, keypoint estimation, depth estimation, and image editing [12, 13, 49, 67-69].
and image generation [9,
3 Method
We propose a novel method for weakly supervised open-vocabulary segmentation of reconstructed NeRF, Given the multi-view images of a scene and the open-vocabulary text description for each class, we aim to segment the reconstructed NeRF such that every 3D point is assigned a corresponding class label.
To achieve this, we exploit the CLIP model's multimodal knowledge by mapping each 3D point to a CLIP feature representing its semantic meaning. As CLIP only generates image-level features, we extract a hierarchy of CLIP features from image patches and learn 3D Selection Volune for pixel-level feature extraction, described in Sec. 3.1.
3

---

Algorithm I: Extructing pixel-level features of an image from CLIP
Input: RGB image I E R number of scale N, CLIP image encoder E, Output: multi-scale pixel-level features F1 E RNxxDsix
Initialize: the patch size of each scale patch_sizes e R , F, zeros, ,, D. W)
j+ Loop over all the scales +/
acale_idze patchpattiiaae si5c in
stride = patch_size/4
count zeros(1, 1, H. W') /> Record the patch count for each pixel +/ Pemslti = zeros(1, D, (D) Multi-spatial feature of current scale +/
Loop for s_ids in raage((H - -patch_size]] sith_ssssssssseeeeeeee +) do
the patchea +/
start _t. = x _iids x stride
for y_idu in range((W
sitthh_ssze//srriide
do
start_y = g_ide x stride
Js Get image patch'a coordinates with randorneaa +/
(lef1, upper, riper, lower) = (max(starl_)) randint(O, stride), max(start_ randint(O), stride), 0).
main(start_@ patch_size randint(O, (ttride W), min(starf_z patch_aise randint(O, stride), H))
/a Get image patch'a CLIP Ieature +/
Fautch E.((L.roo(leet upper, spper, (ight,,oower]
Fmnits spatral i. upper: lower, feft: :rght]] += patch
countfin upper lower, left: right] +
end
Fmulti spatial = cound
Filsccaee idx) Fmoulti Apaahii
end
However, the image patches' CLIP features may have ambiguities for segmentation, showing in- accurate absolute relevancy values which lead to misclassification. Take an image patch capturing an apple lying lawn lawn information. If the apple is relatively small, the patch could be classified as lawn because lawn dominates the patch's 's CLIP features. Then the class apple would be ignored. To address this issue, we introduce a Releevancy-Distriuutton Alignment (RDA) loss, aligning the segmentation probability distribution with each class's s normalized relevancy map, described in Sec. 3.2. For precise object boundaries, we align the segmentation probability distribution with the images' DINO features. Previous segmentation approaches utilizing DINO features have focused on unsupervised segmentation [49, 68]. lacking semantic meaning for segmented parts. In the open- ppen-vocabulary context, assigning accurate text labels segmented regions requires aligning DINO feature clusters with correct text labels. To overcome this challenge, we propose a Skature-Distribbee loss, segmenting the scene based on DINO features' distribution and assigning appropriate text labels, as described in Sec. 3.3.
example. The corresponding CLIP feature contains both apple and
Alignment (FDA)
3.1 Distilling Pixel-level CLIP Features with 3D Selection Volume
We propose a method based on multi-scale and multi-spatial strategies to adapt CLIP's image-level features for pixel-level segmentation, motivated by the observation that a pixel's semantic meaning should remain invariant to its surrounding pixels. The multi-scale component extracts features from patches of varying sizes around each pixel, and the multi-spatial component extracts from patches in which each pixel is at different positions. We average the multi-spatial features in each scale, attributing the primary direction of these features to the pixel's semantic meaning. We utilize sliding-winde window algorithm for multi-sy molti-spatial feature extraction. To prevent checkerboard patterns in the features and potential segmentation artifacts, introduce randomness in the window size. The algorithm's pseudo-code is in Alg. 1.
After extracting pixel-level features from CLIP, now have the multi-view RGB images and their corresponding multi-scale pixel-level features. Each ray r is assigned its multi-scale feature D for supervision. Rather than simply averaging each ray's multi-scale features across scales, we introduce a 3D Selection Volume S determine the most suitable scale indicative of the object size within patch. Each 3D point x E R yields sclection vector S, E IR,, from S. Following [4, 45], we introduce an additional branch to render the CLIP feature. We can then render the RGB value, the CLIP feature, and the selection vector of each ray using volume rendering [6]:
F(r) E DD
C(r) = Σ TTa,CEERR f(r)= = TTaPERRR
(1)

---

Region
Region 2
Region 1
Region
Relevancy
Relevancy
class label
label
@
Ta
Ta
Normalize
Tb
Tb
Spatial Location
Spatial Location
Figure 2: Mitigating CLIP features' ambiguities with normalized relevancy maps. For original relevancy maps Ta. of classes and b, we note higher relevancy for class b in Region 2 than in other image regions. Despite this, the ambiguities of CLIP features lead to Region 2's classification due to the higher absolute relevancy of Region even is located in Region 1. To rectify this, we normalize each class's relevancy maps to a fixed range. These normalized relevancy maps, ra and rb. reduce such ambiguities, facilitating accurate region-class assignments.
Σ TaaS E 10,1
S(r) = Softmax
where C₁ S; are the color, feature, and selection vector each sampled point along the ray, T; = (1 - ax;) the accumulated transmittance and a; = exp(-6,o0 is the opacity of the point. We apply Softmax function to the selection vector of each ray such that the sum of the probability of each scale is equal to I.
For a set of rays R in each training batch, the supervision loss can then be formulated as the combination of the L2 distance between rendered and ground truth RGB values and the cosine similarities cos(,) ) between the rendered features and the selected multi-scale CLIP features:
Σ (I((r)) -(r) cos ((()) FE
Laupervision
Given a set of text descriptions [[CLASS]: of C classes and the CLIP text encoder E,, we can get the classes' text features = Eg ([CLASS] E RCDD, Then we can get the segmentation logits z(r) of the ray by computing the cosine similarities between the rendered CLIP feature and the classes' text features:
(rr) cos (()) ER
We then get the class label of the ray ((r) argmax(2(r)).
3.2 Relevancy-Distrrbuttin Alignment for Ambiguity Mitigation
To mitigate the ambiguities of the CLIP features, we propose to align the segmentation probability distribution with the spatially normalized relevancy maps of each class, enabling method to identify specific image regions described by each class text, as illustrated in Fig. 2. The segmentation probability of each ray P(r) can be derived from the segmentation logits with a Softmax function:
P(r) = Softmax (z(r)) E =00
(5)
The relevancy of a given class is determined by the similarity between the class's text feature and the selected feature from the hierarchy of image patches' CLIP features. Given an image 7, we can get its multi-scale pixel-level CLIP feature Fj E R HxW using Alg. and selection vector S₁ E Nx using Eq. (2). And then we can get the image's relevancy map R₁ E RxxHXX as:
R Sh.. COs(T, Fin..
(6)
where where h w denotes the index in the and W channel. We normalize each class's relevancy independently within input view [0, to mitigate the ambiguities of CLIP features, making our method discern image regions described by each class text:
R1 (Rı min(R;)) (max(R)) min( (R,)) [0, 1xx
(7)
5

---

Sogmenttiinn Probeblity
Equally Diteer
Different
Similes
Equactyy
X Instable
Torget Distribution
Stoble
Figure 3: Difference between similar and distant distributions. Distributions having large di- vergence from the target distribution exhibit significantly diverse shapes, increasing the training instability (left). Conversely, distributions displaying low divergence with the target distribution consistently demonstrate a similar shape (right).
where min() and max() are the functions getting the lowest and highest values across the spatial dimensions (i.e. H and W). We apply Softmax function to Ry to make probability vector. Then we can assign each ray its normalized relevancy with all the classes R(r) E [0, i We employ the Jensen-Shannon (JS) divergence to measure the discrepancy between the normalized relevancy R(r) and the segmentation probability distribution P(r) of each ray, formulating the Sldevancy-Distribuiig Alignment (RDA) loss:
P(r) MMR(T)e) + R(r).log ( R(r)) (MPA()))) )) /2,
CRDA=
=Σ CEC
Por).log
(8)
where (r)(r)) = (P(r) + R(r))/2 is the average of the two distributions, and the subscript c denotes the probability of the eth class. By aligning the normalized relevancies and the segmentation probability distributions, our method can effectively identify the specific region corresponding to the text description of each class.
3.3 Feature-Distributtn Alignment for Precise Object Boundary Segmentation
To ensure the segmentation exhibits precise object boundaries, we align the segmentation probability distribution with the images' DINO features, which have been shown to capture superb scene layouts and object boundary information [12, 13]. Following [49, 68], we extract the scene layout information with DINO feature correlation tensor. Given patch of size Hp x W,, we can get the correlation tensor Corr_F H W. as:
Corr_Fhaij = cos(fww Sey),
whose entries represent the cosine similarity between the DINO features f at spatial positions (h, w) and (i,j) of the patch. In order to construct the correlation tensor for the segmentation probability distribution, we propose utilizing the JS divergence to assess the similarity between segmentation probabilities two distinct spatial positions. The choice of JS divergence offers several ad advantages, including its symmetric nature and bounded range of [0, 1]. which contribute to improved numerical stability. However, since we only care about the class label of each point, i.e. the entry with the highest probability, we use low temperature < to get a sharper version of the segmentation probability distribution P to let the model focus on the entry with the largest probability:
P = Softmax (2/T) [0.
(10)
The distribution correlation tensor Corr E HH,WaxHHW, can thus be computed with:
Phwee
Pije
) /2.
Corr Corr_Dhwri = Σ Powelog
+ Piyclog
(11)
M
M Mppe
cEC
where Phwwey are the segmentation probabilities of the eth class at spatial locations (h, w) and (i,j) of the patch, M = (Phw P.y) the average of the two distributions. Thus the correlation
loss [68] can be expressed as:
Loorr
(Corr_Fhowl b) x Corr Corr_Dhwij hwij
Σ
(12)

---

Table 1: Quantitative comparisons. We report the mloU(1) scores and the Accuracy(1) scores of the following methods in scenes and highlight the best, second-best and third-best scores. Our method outperforms both 2D and 3D methods without any segmentation annotations in training.
Methods
bed
safa
lawn
bench
table
mlot)
Accuracy
Accuracy
Accuracy
intoU
Accuracy
mlot'
Accurncy
mloU
Accuracy
LSeg
87.6
46.1
42.7
2D
ODISE [16]
52.6
86.5
48.3
35.4
39.8
82.5
52.5
59.7
24.1
39.0
39.7
34.5
OV-Seg [19]
79.8
40.4
66.1
69.5
81.2
92.1
71.4
49.1
88.9
89.2
80.5
65.3
FFD [4]
56.6
86.9
03.7
09.5
42.9
82.6
25.1
51.4
06.1
42.8
07.9
30.1
Sem(OD)SF) [45]
50.3
86.5
27.7
22.2
24.2
80.5
29.5
61.5
25.6
56.4
18.4
30.8
Segg)) [45]
89.3
96.7
1.ERF [2]
23.5
86.9
27.0
43.8
73.7
93.5
46.6
79.8
53.2
79.7
33.4
41.0
Ours
89.5
96.7
74.0
91.6
88.2
97.3
92.8
98.9
89.3
96.3
88.8
96.5
where b is hyper-papr syper-parameter denoting that consider the segmentation probabilities of two spatial locations (h, w)) and (i,jj) to be similar if their DINO features' similarity is larger than and distant less than b. Nonetheless, the correlation loss Leorr introduces significant instability due the diverse shapes of distributions with large divergence from a target distribution, making the loss assign wrong labels to the segmented parts. Conversely, when distribution displays low JS divergence with the target distribution, it consistently demonstrates similar shape to the target distribution, as shown Fig. 3. Based on this observation, we propose re-balancing the weights associated with similar and dissimilar DINO features. Specifically, we allocate much greater weight to the correlation loss arising from similar DINO features and a smaller weight to that of dissimilar DINO features, thereby mitigating the instability caused by the correlation loss. Thus the Feature-Diss Distribution Alignment (FDA) loss can be formulated with:
pos_ F- clamp(CCrr__ bb, min 0); neg F clamp(Corr_F max =0, (13)
LFDA Apos
E(pos
Plowis Corr D /cooun__oonzeropps__]]
harij
Aneg
Σ hwnig
(neg (neg_Fnwsk x Corr Corr
(14)
where clamp(, min/max 0) is clamp all the elements smaller/greater than 0 to 0, thus making pos_F 2 0 and neg__F FF < O, count_nonzero() is to count the number of non zero elements, and Apos, Aneg are the weights associated with similar and dissimilar DINO features, which set as Apos 200 >>
Xneg- = 0.2 by default.
The total training loss is: L Lespervision + LRDA + LFDA, where Leupervision CRDA calculated with randomly sampled rays and LFDA is calculated with randomly sampled patches.
Experiments
We evaluate method on 3D open-vocabulary segmentation, showing that our method recognize long-tail classes and produce highly accurate object boundaries even with limited input data. We employ TensoRF [42] as the backbone and extract 3 scales of pixel-level CLIP features. More implementation details and experiments are in the appendix.
Dataset. Existing 3D segmentation datasets predominantly focus on either restricted scenes with narrow range of object classes [5, 70], or individual objects [71-73], thereby limiting their capacity fully assess the task of 3D open-vocabulayy segmentation. Thus following [2], we create a dataset comprising 10 distinct scenes. Each scene features set of long-tail objects situated in various poses and backgrounds. Ground truth masks for the test views are manually annotated, enabling both qualitative and quantitative evaluation of our segmentation methods. We also evaluate our method on more diverse datasets which include human body, human head, indoor scenes with low-quality images [70, 74], and complex scene from LERF datasets [2], as shown in the appendix.
Baselines. We benchmark our method with three NeRF-based methods capable of 3D open- vocabulary segmentation: FFD [4], Semantic-NeRR (abbreviated as Sem) [45], and LERF [2].
7

---

RGB
FFD
LERF
Ours
Ground Truth
View
View 2
shrilling
weaving
View
View
New York cap
headphone
green soap
apple
stapler
View 1
View 2
Gundam
Pikachu
wireless controller
Nintendo Switch controller
of groy
Figure 4: Qualitative comparisons. Visualization of the segmentation results in 3 scenes. Our method successfully recognizes long-tail classes and produces the most accurate segmentation maps.
FFD, pioneering effort in 3D open- vocabulary segmentation, applies the feature maps of the 2D open- voenboocabulary segmentation method, LSeg [8]. to achieve its segmentation results. For Sem, we adapt the open-vocabulary segmentation context by distilling segmentation results from two state-of-the-art 2D open-vocabulary segmentation models: the and the CLIIP-based OV-Seg [19]. LERF closely aligns with our proposed approach due its of knowledge distillation from CLIP and DINO. However, its primary focus is on object localization rather than segmentation. We the same scale level number and patch sizes LERF for fair comparisons. We also include results obtained by independently segmenting each test view using the aforementioned 2D models. Note that under settings, FFD, Sem fully supervised methods using segmentation annotations.
ODISE [16],
4.1 Results
We present the qualitative results in Fig. 4 and quantitative results in Tab. 1. Our proposed method outperforms all other techniques, including those which heavily rely on extensive segmentation annotations, such as LSeg, ODISE, OV-Seg. In particular, ODISE and FFD underperform in our evaluation, as they are unable to identify many long-tail tail classes, suggesting that the fine-tuning process of L.Seg and ODiSE may have led a significant loss of the open-vocabulary knowledge originally encapsulated by CLIP and Stable Diffusion [23]. OV-Seg attempts retain CLIP's knowledge by leveraging mined dataset, however, requires mask proposal model which produces inconsistent segmentation across views, making Sem(OV-Seg) produce noisy and imprecise segmentation. LERF

---

Ablations
Limited Input
RGB
Full Model
w/o re balance
50% Views
10% Views
Ground Truth
w/o RDA
w/o FDA
Single Scale
Single & 10%
dressing doil Portuguese egg
wood
orange cat affread
pebbled concrete wall
Figure 5: Studies. Visualization of the studies ablations and limited input.
also fails to capture precise object boundaries due its usage relatively naive regularization loss, which fails to fully exploit the object boundary information within the DINO features. In contrast, our method exhibits robust performance, successfully recognizing long-tail classes and generating accurate and well-defined boundaries for each class. However, LERF allows querying any object without the need for running optimization again, which is an advantage over our method.
4.2 Studies
Ablations. We conduct ablation studies to evaluate the individual contributions of the RDA loss and the FDA loss the overall perfor- mance of our proposed method. As shown Tab. 2, both RDA loss and FDA loss crucial result in severe performance degradation. As illustrated Fig. 5, without the RDA loss, the model does resolve the ambiguities of the CLIP features, leading to misclassifications. For instance, it fails distinguish between orange and Portuguese egg lart, and confuses mini offroad car with wood. Without the FDA loss, al- though our method can correctly locate each class, fails segment precise object boundaries. When discarding the re-balancing in the FDA loss, i.e. using the correlation loss [68]. the model produces accurate boundaries but assigns each cluster the wrong label due to the instability brought by diverse distribution shapes.
Table Studies. We report the mean mloU(1) scores and the Accuracy(f)) scores in our studies.
method, without each of which
mloU
Accuracy
w/o RDA loss w/o FDA loss
$7.2
79.4
58.2
82.7
w/o re- re-balance
44.9
74.3
50% views
85.7
95.7
10% views
79.1
94.6
single scule
85.2
95.5
single & 10%
77.1
94.6
model
Limited input. Given the substantial computational and storage demands of extracting hierarchical CLIP features for each view (exceeding 1GB for 3 scales the full model), we explore whether reducing input CLIP features would yield similar results, shown in Tab. 2 and Fig. 5. We test two modifications: reducing views for feature extraction and using single scale of CLIP features rather than three. Halving the input views for feature extraction leads to negligible performance degradation (< 1%) and minor visual differences compared to the full model. When reducing to only 10% of input views, equivalent to 2-3 views in our dataset, we observe a modest 9% drop in the mloU score and 1% decrease in the Accuracy score, while retaining accurate segmentation across most classes. Using a single scale of CLIP features also only incurs minimal degradation (< 1%). Even under extreme conditions, i.e., extracting a single scale of features from 10% input views (total only 1/30 input of the full model), performance degradation is limited to 10%. This efficient approach even outperforms LERF [2] which utilizes input views and scales, highlighting our method's robustness.
5 Limitations
The limitations of our method are twofold. First, unlike LERF [2], our method requires text labels before training. To perform segmentation with text labels, our method needs to be retrained.

---

Inferring accurate boundaries with open vocabularies is challenging for implicit representations like NeRF, as NeRF learns continuous representation rather than a discrete one. is promising to learn object-level discrete representation using NeRF in future work.
Second, since method has never seen any segmentation maps during training (only weakly supervised by the text labels), it fails to segment complex scenes like indoor datasets [70, 74] with high precision, as shown in the appendix. Our method distills pixel-level CLIP features in a patch- based fashion with strong inductive bias for compact objects with an aspect ratio close to 1. For objects with large complex shapes and unobvious textures, our method would fail recognize them.
Conclusion
In this study, address the challenge of 3D open-vocabulary segmentation by distilling knowledge from the pre-trained foundation models CLIP and DINO into reconstructed NeRF in a weakly supervised manner. We distill the open-vocabulary multimodal knowledge from CLIP with Selection Volume and a novel features. In addition, we introduce a novel Alignment loss to mitigate the ambiguities of CLIP Alignment loss to extract accurate object boundaries by leveraging the scene layout information within DINO features. Our method successfully recognizes long-tail classes and produces precise segmentation maps, even when supplied with limited input data, suggesting the possibility of learning 3D segmentation from 2D images and text-image pairs.
Acknowledgement
We sincerely thank Zuhao Yang, Zeyu Wang, Weijing Tao, and Kunyang Li for collecting the dataset. This project is funded by the Ministry of Education Singapore, under the Tier-1 project scheme with project number RT18/22.
References
[1] Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam. Clip- fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663 2022.
[2] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXivv2303.0955 2023. 7. 16, 17
[3] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.00 230.::23300.02211, 2023. 1,
[4] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. arXiv preprint arXiv:2205.J 15585, 2022. 1, 7, 17
[5] Angela Dai, Angel Chang, Manolis Savva, Maciej Halher, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotater 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017. 1,
[6] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99-106, 2021. 1, 3,4
[7] Songyou Peng. Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys. Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211. 2022.
3.
[8] Boyi Li. Kilian Q Weinberger. Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint urXiv:2201.03556 2022. 3. 7,8
[9] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-img image pretraining, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793- 16803, 2022. 3,3
[10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from
10

---

natural language supervision. In International conference on machine learning, pages 3748-8763. PMLR,
2021. 2,3
[11] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarezyk, Clayton Mullis, Aarush Katta, Theo Caombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 14, 2021. 2,
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e e J' l'ego., Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sell-supervised vision transformers. 2021 IEEE/CVF International
Conference on Computer Vision (ICCV), pages 9630-9640, 2021. 2, 3,
[13] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023. 2, 3,
[14] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with Icarnable centers for open-vocabulay semantic segmentation. arXiv preprint arXiv:221 1.122/,,.1481,, 2022.
[15] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[16] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv: 2023. 7.8
[17] Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.112700 2022.
[18] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583- 11592, 2022.
[19] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang. Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapte clip. arXiv preprint NXV:22210004150, 2022. 3, 7,8
[20] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021. 3
Sung,
[21] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive sagguage--mage pre-training paradigm. arXiv preprint arXiv:2/ 10.05208, 2021.
[22] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 529-544. Springer, 2022.
[23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 3,
[24] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y- Yi Lin. Scaling opcablocybulary image segmentation with image-level labels. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557. Springer, 2022. 3
[25] Tro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv: 0002007020100 2017. 3
[26] Jens Behley, Martin Garbade. Andres Milioto, Jan Quenzel, Sven Behnke. Cyrill Stachniss. and Jurgen Gall, Semantickitti: A dataset for semantic seene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9297-9307, 2019.
[27] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-1163)
2020. 3
[28] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint (rXXiv17099066588 2017. 3
[29] Dave Zhenyu Chen, Angel X Chang. and Matthias NieBner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part XX, pages 202-221, Springer, 2020.
11

---

[30] Andreas Geiger, Philip Leaz. and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361.
IEEE, 2012.
[31] Binh-Son Hua. Quang- Quang-Hieu Pham, Due Thanh Nguyen, Minh-Khoi Tran, Lap-Fat Fai Craig Yu, and Sai- Kit Yeung. Scenenn: scene meshes dataset with annotations. 2016 Fourth International Conference on 3D Vision (3DV), pages 92-101, 2016.
[32] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45: 3292-3318 2021.
[33] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 909-918, 2019.
[34] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognation, pages 2446-2454, 2020.
[35] Lyne P. Tehapmi, Christopher Bongsoo Choy, Ito Armeni, JunYoung Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. 2017 International Conference on 3D Vision (3DV), pages 537-547. 2017.
[36] Loie Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4558-4567, 2018.
[37] Runyu Ding. Jihan Yang. Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Language- driven open- open-vocabulary scene understanding. arXiv preprint vxXiv22211.63311 2022. 3
[38] Kirill Mazur, Edgar Sucar, and Andrew J Davison. Feature-rcalistic neural fusion for real-time, open set seene understanding. arXiv preprint arXiv:2210.03043, 2022.
[39] Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision- language models. In Conference on Robot Learning, 2022.
[40] Rumnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efticiaa 3d scene understanding by clip. arXiv preprint arXiv:2301.04926, 2023. 3
[41] Iro Armeni, Ozan Sener, Amir Roshan Zamir, Helen Jiang, Ioannis K. Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1534-1543, 2016.
[42] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, 2022. 3, 7, 15
[43] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. arXiv preprint arXiv:2212.13545 20212,2222222.....2 3
[44] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-superviser 2d image representations. 2022 International Conference on 31) Vision
(3DV), pages 443-453, 2022.
[45] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15818-15827, 2021. 3,4,7
[46] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. arXiv preprint arXiv:22 09.822 2022.
[47] Vadim Tschemezzi, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. 2021 International Conference on 3D Vision (3DV), pages 910-919, (00-919.0211. 3
[48] Jesus Zarzar, Sara Rojas, Silvio Giancola, and Bernard Ghanem. Segnerf: 3d part segmentation with neural radiance fields. arXiv preprint 1rXiv:2211.11215. 2022.
[49] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view
self-supervised object segmentation complex scenes. arXiv preprint arXiv:2209.0877 087777 2022. 3, 4,
[50] Shengnan Liang, Yichen Liu, Shangzhe Wu, Yu-Wing Tai, and Chi- Keung Tang, Onerf: Unsupervised 3d object segmentation from multiple views. arXiv preprint 2XX::22211111056 2022. 3
12

---

[51] Karl Stelzner. Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsuper- vised volume segmentation. arXiv preprini arXiv:2104.01148 2021. 3
[52] Rishi Bommasani, Drew A Hudson, Ehsan Adcli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021. 3
[53] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji, Qihen Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302. 09939, 2023. 3
[54] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models
are unsupervised multitask leameer. OpenAl blog, 2019. 3
[55] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamners. Advances in neural information processing systems, 33:1877 1901, 2020.
[56]
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[57]
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprini 1xX0::811000400: 2018. 3
[58] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zeutlemoyer and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020. 3
[59] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequencc pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.1 134611 2019. 3
[60] Kevin Clark. Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint 2rXiv:2003.10555 2020.
[61] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. Al- bert: A lite bert for self-supervised learning of language representations. arXiv preprint 1KKv:1999.1199 2019. 3
[62] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv: 06125, 2022. 3
[63] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information P Processing Systems, 35:36479-36494 2022. 3
[64] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022. 3
[65]
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11686-11666 11666, 2022. 3
[66] Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. Clip models few-shot learners: Empirical studies on vqa and visual entailment. In Anmual Meeting of the Association for Computational Linguistics, 2022. 3
[67] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10738- 10747, 2022. 3
[68] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation hy distilling feature correspondences. arXiv preprint 2rXiv:2203.08414 2022. 4, 6,
[69] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2/ 2(3):4, 2021. 3
[70] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint
2019. 7, 10, 17
[71] Matt Deitke, Dustin Schwenk. Jordi Salvador, Luca Weihs, Oscar Michel, Bli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08000 2022,
2022.
13

---

[72] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan. Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. arXiv preprint urXiv:2301.07522 2023.
[73] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of IEEE/CVF International Conference Computer Vision, pages 10901-10911, 2021.
[74] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of IEEE/CVF international conference on computer vision, pages 10912-10922, 2021. 7, 10, 17
[75] Johannes Lutz Schönberger and Jan-Michael Frahm. Strmmtrr-frrmmmmttiin revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[76] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines, ACM Transactions on Graphics (TOG), 2019.
14

---

Appendix
This document provides supplementary materials for Weakly Supervised 3D Open-vocabulary Seg- mentation in implementation details (Appendix A), more ablations (Appendix B), more evaluations (Appendix C), and more results (Appendix D).
A Implementation Details
A.1 Model Architecture
Shared Volume
MLP
CLIP Feature
MLP
RGB
View Direction
Density Volume
Selection Volume
Density
Selection Vector
Figure 6: Model architecture.
We use TensoRF [42] as our base NeRF architecture for efficiency, the plane size is also the same as the default setting of TensoRF. The RGB and CLIP feature branches share the same volume and use the same intermediate features. The selection volume and density volume are two other independent volumes. We directly use the features extracted from the selection volume and density volume as the selection vector and the density value, as they have low dimensions and are wew-independent. We use the original MLP architecture in TensoRF to extract the view-dependen RGB value and use another MLP which discards view direction input to extract the rendered CLIP feature. The architecture is illustrated in Fig. 6.
A.2 Hyperparameters
We set = 0.2 to get the shaper segmentation probability distribution P. The offset b is set to 0.7 to measure the similarities of the DINO features, meaning that two DINO features are considered similar if their cosine similarity is larger than 0.7, and different if less than 0.7. We use 3 scales of CLIP features, and the patch sizes each scale are set as 8/5, s/7, and s/10, where is smaller value in the width and height of the input image /. In the ablation studies, we use s/7 as the single-scale scale CLIP feature input. The weights associated with similar and dissimilar set Xpos 200 and Aneg 0.2 by default. In certain scenes, find that setting Aneg to 0.22 or 0.18 can produce better results. We use ViT-B/16 CLIP model to extract the image and text features and use version I dino, vitb8 because it employs the smallest downsampling factor of 8 which is advantageous for high-precision segmentation.
patch size
DINO features in LFDA
_itb model to extract the DINO features
A.3 Training
To reconstruct a NeRF from multiview images of scene, we follow the same training settings as TensoRF, For segmentation training, train the model for 15k iterations. In the first 5k iterations,
15

---

Table Dataset. We list the collected 10 scenes and the corresponding text labels. The background
labels are in Italic font.
Scene
Text Labels
red bag, black leather shoe, banana, hand, camera, white sheet
bed
stack of UNO cards, red Nintendo Switch joy-con controller, Pikachu, Gundam, Xbox wireless controller, grey sofa
sofa
red apple, New York Yankees cap, stapler, black headphone, hand soap, green lawn
lawn
shrilling chicken, weaving basket, rabbit, dinosaur, baseball, wood wall
room
Portuguese egg tart, orange cat, green grape, mini offroad car, dressing doll, pebbled concrete wall, wood
bench
a wooden ukulele, a beige mug, GPU card with fans, a black Nike shoe, a Hatsune Miku statue, lime wall
table
the book of The Unbearable Lightness of Being, can of red bull drink, white keyboard, a pack of pocket tissues, desktop, blue partition
office desk
a bottle of perfume, sunglasses, a squirrel pig doll, JBL bluetooth speaker, an aircon controller, blue-grey sofa
blue sofa
Coke Cola, orange juice drink, calculator, pitaya, Glico Pocky chocolate, biscuits sticks box, desktop
snacks
Winnie-the- Dove body wash, gerbera, electric shaver, canned chili sauce, desktop
covered desk
freeze the shared volume and density volume, and train the selection volume and the CLIP feature branch. For the rest 10k iterations, we further finetune the shared volume and the RGB branch. We Adam optimizer with belas (0.9, 0.99). The learning rates for training the volume and MLP branch are respectively set to 0.02 and 1e 4. For finetuning the volume and the MLP, the learning rates set to 5e and 5e 5. We also employ learning rate decay with factor of 0.1.
The multi-scale pixel-level CLIP features of training views are pre-computed before training and the DINO features are computed with sampled patches on the fly during training. When computing Csupereision and CRDA, we randomly sample rays with a batch size of 4096. When computing LFDA randomly sample patches of size 256 x 256 with batch size of 8. We use downsampling factor of 8 when sampling rays and a factor of 5 when sampling patches. The model is trained on an NVIDIA A5000 GPU with 24G memory for ~ 1h30min for each scene.
A.4
Dataset
We capture 10 scenes using smartphones and use Colmap [75] to extract camera parameters for each image. We capture 20 ~ 30 images for each scene and the resolution of each image is 4032 x 3024. We follow the data structure LLFF [76]. We manually annotate the segmentation maps of views for each scene as the ground truth for evaluation.
We list the text labels used our experiments in Tab. 3. Note that we sometimes choose to add a general word to describe the whole background, such as wall, desklop, following LERF [2]. The text labels in our dataset contain many long-tail classes, which can be used to fully assess the open- vocabulcyyy capability.
16

---

RGB
Ground Truth
Full Model
w/o neg_
w/o Selection
dressing doll
green grape
wood
orange mini offroad car
Portuguese egg tart
pebbled concrete wall
Figure 7: More ablations.
View
View
head
eyes
Figure 8: Evaluation human body dataset (left). Evaluation human head dataset (right).
B More Ablations
We perform two more ablation studies on the Selection Volume and the FDA loss, shown in Tab. 4 and Fig. 7. Without the Selection Volume, we simply average the multi-scale CLIP features rather than learning the appropriate scale. We can see that both the mloU and the Accuracy score are inferior to the full model. We could discard the dissimilar part neg_F F since dissimilar DINO features often impair the stability of the correlation loss. However, neg F encourages different segmentation probabilities for different seman- tic regions and it plays crucial role for precise object boundary extraction.
Table 4: More Ablations.
mloU
Accuracy
w/o neg F w/o Selection
76.9
92.4
84.8
95.3
full model
86.2
C More Evaluations
We additionally perform evaluations on human body, human head, indoor datasets with low-quality images [70, 74], and complex scene from LERF datasets [2]. We compare with the concurrent work LERF qualitatively due to the lack of labels or the defective annotations as pointed out in [4]. We also perform experiments with different text prompts. We use the same scale level number and patch sizes in all comparisons.
Human body and head. As shown in Fig. 8, our method segments more precise parts than LERF Specifically, LERF fails to segment the "head" in the human body and the "black T-shirt" in the human head. In contrast, our method can recognize and segment these parts correctly because our designed RDA loss addresses the ambiguity of the CLIP features effectively.
Indoor scenes with low-quality images. Fig. 9 shows experiments on the indoor datasets [70, 74], where many images are unrealistically rendered with less photorealistic appearances (as indicated in [4]) and have limited spatial resolution (640 x 480 or 1024 768). Due these data constraints,
17

---

LERF
View
View
groy
coflee
cushion
white
window
carpet
Figure 9: Evaluation indoor datasets with lower quality images.
ouTs
View
100
glass
Apple
with
showp
blue partition
dekktop
red
coffee mug
concrete ground
paper nopkón
plate with cookies
Figure 10: Evaluation on a complex scene from LERF datasets (left). Evaluation on a scene with multiple instances of a same class (right).
our method sometimes confuses labels with similar appearances. However, we can see that our method still outperforms LERF by successfully segmenting more labels.
Complex scenes. Fig. 10 (left) shows the segmentation of challenging sample from the LERF dataset, where the scene has complex geometry as well as many objects of varying sizes. It can be observed that LERF cannot segment most objects due the ambiguities of CLIIP features while method can segment more objects correctly with more precise boundaries. Fig. 10 (right) shows a scene with multiple instances of appearance, texture, etc., they also have similar DINO features. As a result, FDA will not mistakenly segment them into different classes. The RDA loss will further help by assigning all these instances to the same text label. In the experiment, we observed that our method successfully segments all three apples into the same class with accurate boundaries.
class. Since instances of the same class often share similar
ours-original
ours-rephrased
ours-rephrased
View
View
Portuguese
dressing doll barate
black hewdphone Isten
Figure 11: Evaluation rephrased texts.
18

---

Segmentation with different text prompts. We conduct experiments to segment scenes with different text prompts. In the experiments, we replaced the original texts with different languages (e.g., Portuguese egg tart -> Pastel de Nata), names (e.g.. dressing doll -> Barbie), and actions (e.g.. hand soap -> wash hand, black headphone -> listen to music). As Fig. shows, with the rephrased text prompts, our method can still segment the scenes reliably. The experiments are well aligned with the quantitative experiments as shown in Tab. 5.
Table 5: Evaluation on rephrased texts.
mloU
Accuracy
mloU
Accuracy
original rephrased
88.2
97.3
89.3
96.3
89.3
97.2
88.4
96.6
D More Results
We show more segmentation visualizations our method Fig. 12 (bed), Fig. (sofa), Fig. 14 (lawn), Fig. 15 (room), Fig. 16 (bench), Fig. 17 (table), Fig. 18 (office desk), Fig. 19 (blue sofa), Fig. 20 (snacks), and Fig. (covered desk). The quantitative results are listed Tab. 6.
Table 6: Quantitative results.
bed Accuracy
sofa
lawn
room
bench
mloU
mloU
Accuracy
mloU
Accuracy
mloU
Accuracy
mloU
Accuracy
89.5
96.7
74.0
91.6
88.2
97.3
92.8
98.9
89.3
96.3
table Accuracy
office desk
blue sofa
snacks
covered desk
mloU
mloU
Accuracy
mloU
Accuracy
mloU
Accuracy
mloU
Accuracy
88.8
96.5
91.7
96.2
82.8
97.7
95.8
99.1
88.6
97.2
19

---

RGB
Ground Truth
banana
black leather shoe
camera
hand
red bag
white sheet
Figure 12: bed.
Ours
Ground
Gundam
Pikachu
red Nintendo Switch joy-con controller
Xbox wirelless controller
stack of UNO cards
grey sofa
Figure 13: sofa.
20

---

Ground
Yankees cap
headphone
green
hand soap
red apple
stapler
Figure 14: lawn.
RGB
Ours
Ground Truth
baseball
dinosaur
rabbit
shrilling chicken
weaving basket
wood wall
Figure 15: room.
21

---

RGB
Ground Truth
dressing doll
green grape
wood orange cat
mini offroad car Portuguese egg tart
pebbled concrete wall
Figure 16: bench.
RGB
Ours
Ground Truth
a GPU card with fans
Hatsune Miku statue beige mug
black nike shoe
wooden ukulele
lime wall
Figure 17: table.
22

---

RGB
Ours
Graund Truth
can red bull drink
a pack pocket tissues
white keyboard
blue partition
desktop
the book of The Unbearable Lightness of Being
Figure 18: office desk.
Ours
Ground
JBL bluetooth speaker
bottle of perfume
squirrel pig doll sunglasses
aircon controller
blue-grey sofa
Figure 19: blue sofa.
23

---

RGB
Ours
Ground Truth
Coke Cola
Glico Pocky chocolate biscuits sticks box
calculator
desktop
orange juice drink
pitaya
Figure 20: snacks.
RGB
Ground Truth
Dove body wash
Winnie-the-Ponn
canned chili sauce
desktop
electric shaver
gerbera
Figure 21: covered desk.


Reviews:
[{"review_id": "Ii9xEtWKuw", "reviewer": "Reviewer_1C3c", "summary": "This paper trained a NeRF model leveraging on two self-supervised trained models CLIP and DINO. By distillating multi-view image features from these two large models, the NeRF model is able to synthesize accurate object segmentation masks on unseen scenes with long-tailed distributions. Experiments on 10 self-collected scenes shows that high-quality segmentation masks can be predicted after per-scene training without explicit segmentation annotations.", "questions": "Please see weaknesses above where my main concern is the lack of analysis on diverse popular real-world scenes to really prove the model's capability.\n\nOverall this paper is attempting to tackle a challenging task and I think more discussion could better  strengthen the contributions.", "limitations": "Limitations and potential negative societal impacts have been discussed in the supplement.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a easy-to-follow pipeline to reach accurate open-world 3D segmentation by taking advantages of two pretrained large models CLIP and DINO without access to segmentation ground truth masks.\n\n- The experiments on self-collected scenes prove the capability of proposed method, and the ablations cover most of key design choices made in the paper, with adequate implementation details.", "weaknesses": "- Limited experiments on popular benchmarks.\n\n    My main concern lies on the limited evaluation as 'open world 3d segmentation' is a challenging topic. \n    - In this paper, 10 scenes with multi-view high-res images are taken for experiments. However, more diverse results on different datasets are expected to really prove the capability of ‘open-world 3D segmentation’. Existing datasets like ScanNet, VG150 still contains imbalanced long-tail distributions. How the proposed methods work on ScanNet, for example, where captured images are in VGA resolution and lower quality compared to used data in the paper.\n   - There are many hyper-parameters need to be considered during training. Are they sensitive to difference scenes/datasets?\n\n    Otherwise, the overall experiments may look too curated to prove the main statements.\n\n\n - One question to ask is whether the model sacrifice the ‘close-world’ capability to complement the open-world one. Therefore, it would be good to see how trained methods work on different categories with different types of text prompts.\n\n\n- What is the impact of text descriptions? Are the given text carefully chosen or some general text input covering key information are good enough for generating promising labels?\n\n\nOther comments:\n- Why only 6 of 10 scenes are shown for numerical experiments with baseline methods and 4 scenes are used for ablations? I see a complete 10 scene results of proposed methods in the supplement but not of baselines.\n- One question rather than weakness. Could authors explain why to choose to learn CLIP features with a shared TensorRF volume with colour instead of separately. Is there any specific reason or ablations for such design choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and pointing out we are attempting to tackle a challenging task. Below please find our clarification regarding your comments.\n\n1. **Limited experiments.** Please refer to the **General Response 1 and 2**. We perform additional experiments on indoor datasets as well as other datasets.\n\n2. **Hyperparameters** We would like to clarify that our method is not sensitive to the hyperparameters in most evaluated scenes. Specifically, we use the same hyperparameters in all experiments, except the *office desk* scene where we set $\\lambda_{neg}$ to 0.35 which we found gives the best results. Using the default parameter setting, the score of this scene is mIoU=76.3, mAP=90.1, which still outperforms LERF as shown in Table 1 below.\n\n3. **Text prompts.** Please refer to the **General Response 2**. We provide quantitative and qualitative results with more diverse text labels.\n\n4. **More results.** We conducted experiments on part of the datasets due to limited computation resources. As suggested, we conducted experiments on the remaining scenes and reported the quantitative results of baselines of all scenes in Table 1. We also reported the core ablations about RDA and FDA in all scenes in Table 2. We will include these new experiments in the updated manuscript. Thank you for your suggestion.\n\nTable 1. Results of baselines on the remaining scenes.\n| scene | *blue sofa* || *covered desk* || *office desk* | | *snacks* ||\n|-|-|-|-|-| -| -| -| -|\n|| **mIoU** | **mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|\n| LSeg[3]      |18.7|68.8|19.3|77.0|15.3|36.4|27.1|81.3|\n|OV-Seg[4]     |74.8|88.5|73.7|53.7|85.5|74.8|75.3|46.1|\n|ODISE[5]      |55.2|47.6|63.8|72.2|61.1|69.9|33.6|63.7|\n| FFD[2]       |18.0|74.5|22.6|79.8|20.2|42.0|34.0|85.5|\n|Sem(ODISE)[1] |24.0|38.9|50.0|75.1|69.7|86.2|29.6|61.6|\n|Sem(OV-Seg)[1]|80.8|92.2|71.1|89.1|95.6|98.6|75.7|90.6|\n|LERF[6]       |30.9|40.0|74.5|91.1|55.6|77.0|61.0|85.2|\n| **Ours**     |82.8|97.2|88.6|97.2|91.7|96.2|95.8|99.1|\n\n\nTable2. Results of core ablations on all scenes.\n| scene | *table* || *bench* || *sofa* | | *room* ||*blue sofa*||\n|-|-|-|-|-| -| -| -| -| -| -|\n|| **mIoU** | **mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|\n| w/o RDA      |60.1|36.4|52.4|89.6|39.0|39.2|73.4|97.4|44.7|93.3|\n|w/o FDA       |56.7|76.9|56.7|81.9|47.5|76.3|45.5|78.9|52.2|83.4|\n|w/o re-balance|15.8|53.4|22.1|55.4|19.9|69.0|22.5|56.5|52.3|83.4|\n|**ours**      |88.8|96.5|89.3|96.3|74.0|91.6|92.8|98.9|82.8|97.7|\n\n\n| scene | *coverd desk* || *bed* || *lawn* | | *office desk* ||*snacks*||\n|-|-|-|-|-| -| -| -| -| -| -|\n|| **mIoU** | **mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|**mIoU**|**mAP**|\n| w/o RDA      |70.7|91.5|88.6|96.8|56.9|94.0|22.0|64.3|64.3|92.0|\n|w/o FDA       |66.9|88.8|61.3|80.1|64.0|89.3|66.3|83.0|65.1|88.3|\n|w/o re-balance|67.1|88.9|61.2|80.0|63.7|89.2|62.3|83.0|63.0|88.5|\n|**ours**      |88.6|97.2|89.5|96.7|88.2|97.3|91.7|96.2|95.8|99.1|\n\n5. **Shared Volume.** This design follows Semantic-NeRF [1] and FFD [2], which show that using the same intermediate features for both the RGB branch and the feature branch can lead to better results in lifting 2D features to 3D.\n   \n[1] In-Place Scene Labelling and Understanding with Implicit Scene Representation\n\n[2] Decomposing NeRF for Editing via Feature Field Distillation\n\n[3] Language-driven semantic segmentation\n\n[4] Open-vocabulary semantic segmentation with mask-adapted clip\n\n[5] Open vocabulary panoptic segmentation with text-to-image diffusion models.\n\n[6] LERF: Language Embedded Radiance Fields"}, {"review_id": "IHwzBRzq04", "reviewer": "Reviewer_t9cD", "summary": "This work addresses the open vocabulary segmentation by leveraging pretrained foundation model, DINO and CLIP. They firstly lifts 2D CLIP features into NeRF. Then, relevancy distribution alignment loss is used to mitigate the CLIP features's ambiguity. Feature distribution alignment loss is used to improve object boundaries from DINO feature. ", "questions": "See weaknesses.", "limitations": "The unclear parts in the paper and the limited experiments jointly make the performance of this work not convincing enough.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "3D open vocabulary segmentation is an interesting and impactful topic. It is very challenging to use a fully supervised way for training such a model since annotation costs large amount of human labor. This paper leverages 2D foundation models. The presentation is fairly good and the proposed relevancy distribution alignment and feature distribution alignment losses are well motivated.\n\n", "weaknesses": "There are many unclear parts that make me feel hesitated to suggest acceptance of this paper.\n\n1. how they used CLIP is not well justified. Will the patch-wise method break the CLIP feature space? I assume many patches covers only background or small portion of objects.\n\n2. Why DINO? CLIP and DINO are two most important models for this work. So, motivating the utilization of them is important. Especially for DINO, why it is good for object boundaries. Visualization would help or ablation study will help.\n\n3. the most concerning part is the experiments. The method is validated in limited settings. The class set is small that is not enough to show \"open vocabulary\" capability. The dataset is mall. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Below please find our clarification regarding your comments.\n\n1. **CLIP features.** As many patches cover only the background or a small portion of objects, we use multi-scale CLIP features with varying patch sizes to mitigate this problem. The Selection Volume can be learned to pick sensible patch sizes to yield optimal segmentation. It is important to note that the patch-wise feature extraction is used in the concurrent LERF [1] as well, which achieves good results in recognizing objects using open-vocabulary text.\n\n\n2. **Why DINO.** DINO demonstrates excellent performance in unsupervised segmentation [4] due to its ability to encode powerful, well-localized semantic information at high spatial granularity. Previous methods [2, 3] have shown that DINO can segment objects, parts, etc., with accurate boundaries. Additionally, DINO is well-suited for 3D segmentation because it produces continuous features across different views, as opposed to discrete segmentation maps. In contrast, 2D segmentation models often produce inconsistent segmentation maps across views, leading to the presence of artifacts in the final 3D segmentation, as depicted in Figure 4 of the main paper. Please refer to [2] for comprehensive visualizations of the DINO features.\n\n\n3. **More experiments.** Please refer to the **General Response 1 and 2**. We perform additional experiments on more datasets and settings.\n\n[1] LERF: Language Embedded Radiance Fields\n\n[2] Deep ViT Features as Dense Visual Descriptors\n\n[3] NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes\n\n[4] Unsupervised Semantic Segmentation by Distilling Feature Correspondences"}, {"review_id": "4uW68MRfwI", "reviewer": "Reviewer_kCD4", "summary": "This paper tackles the task of 3D scene segmentation using CLIP and DINO distilled into a NeRF representation. Since CLIP and DINO are pre-trained, the method works in an unsupervised fashion i.e., no additional per-point training labels are needed. This is achieved by applying CLIP to small crops in a sliding window fashion resulting in pixel-aligned features. This is repeated for multiple scales, resulting in multiple pixel-aligned CLIP features maps, one for each scale. The paper introduces two new losses (FDA and RDA) that rely on a closed-set of classes during training of the NeRF. The method is evaluated on their own dataset and annotations, and compares to multiple, recent methods (LSeg, ODISE, OVSeg, LERF, FFD).", "questions": "How is the selection vector S_x obtained? Eq. (2) shows that we can render it from the optimized NeRF, that is clear. However, we also need it already during the optimization, see Eq. (3). How does that work, i.e., how can the value S(r) that is only obtained via rendering of the optimized NeRF at the same time be used to optimize the NeRF?\nFinally, I do not see an ablation study that shows the benefits of the 3D Selection Volume. Does the 3D selection volume improve over simply averaging multi-scale features as described in l.151?\n\nMinor:\n\n- l.4 “lack of [...] open-vocabulary segmentation datasets” I would even argue that open-vocabulary datasets are inherently impossible as it would require exhaustive labeling of all possible queries.\n\n- l.7 ”finetuned with close-vocabulary datasets” this is only the case for a subset of visual-language models such as LSeg, other approaches are not fine tuned on closed sets of data e.g. OpenSeg.\n\n- l.35 same point as before: OpenSeg is not fine tuned on a closed set of classes but weakly supervised via image captions.\n\n- l.46 - ‘ambiguous similarities with text description’ what does that mean?\n\n- l.53-55 I don’t understand this sentence, can you break it down for me? What are CLIP feature ambiguities? What is a segmentation probability distribution? What are class relevancies? Until l.62 is similarly unclear - I think it’s mostly unclear because the used terminology is not properly introduced / defined. I can guess what is trying to be achieved but that is not my job, it should be clearly written.\n\n- l.126 “ambiguous similarities” - what does that mean?\n\n- l.127 what are “relevancy values”? Fig.2 is not that helpful since it also does not explain the terms. Again, what is a segmentation probability distribution (l.130)? I can sort of guess it (maybe I’m wrong, maybe I’m right?), why not clearly define and explain these terms since a large part of the main contribution depends on them?\n\n- l.169 Still unclear meaning of “ambiguities of the CLIP features”", "limitations": "Limitations are not addressed. For example, one could discuss the impact of the closed-set of classes C required to optimize the NeRF since this has implications on the practical application (see under weaknesses).", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "A novel “volume-selection” scheme used to align per-image CLIP features to per-pixel CLIP features, in addition to two new losses: Relevancy Distribution Alignment loss and Feature Distribution Alignment loss that do not require a dedicated pixel-aligned CLIP method.", "weaknesses": "I struggle a bit with the intermixing of the term “open-vocabulary” and the fact that the method relies on a known / closed set of classes C. E.g., “open-vocabulary text description for each class” (l.119) somehow implies that the set of classes that we want to segment is known a priori, whereas open-vocabulary implies that there are no such restrictions.\nImportantly, the two losses presented in Sec.3.2 and Sec.3.3 seem to rely on a predefined set of classes C - i.e. **each NeRF is actually optimized for a closed-set of classes, and if the classes change, the NeRF needs to be retrained**. The method can be seen as open-world in the sense that there is no inherent limitation to the classes, however an optimized NeRF is a closed-set representation. Is that correct? I think this aspect should be discussed in more detail in the limitations.\n\nThe writing could be improved, numerous aspects are still unclear and should be properly introduced and explained (Please see the minor points under Questions - I consider them minor since it can be improved in an updated version, however there are so many points that are unclear that the overall presentation of the method is limitted). \n\nThis paper follows an unfortunate trend that is observable across the field of LLM-based methods for open-set scene understanding: each paper presents a new method and at the same time introduces a new (relatively small) evaluation dataset on which they improve over prior/concurrent work. It is clear that such evaluations are not entirely convincing since the results could be cherry picked and favoring the newly proposed model. This paper is unfortunately no exception to that trend. In particular, the dataset selection is motivated by the fact that ScanNet [5] is too limited in terms of classes - however there was a recent extension to ScanNet200 that should mitigate this problem. Also Replica [70] does feature 50-80 semantic classes (depending on the scenes) which shows a long-tail distribution. Hence, there are existing datasets (maybe not perfect) but at least established ones and certainly not worse than very few newly recorded scenes. Further, since the proposed method compares with LERF, it could also compare on the LERF dataset.\n\nThe argumentation / motivation in l.28-31 is unfortunate: it argues that the major challenge of open-set segmentation is the lack of datasets, however that is not really the case since open-set datasets cannot exist as it would require an infinite amount of all potentially existing labels.\n\nThe contributions (1) (in line 71-73) - how does it differ from Decomposing Nerf [4] ? They also use CLIP and DINO in a very similar fashion. Instead of LSeg one could use the proposed multi-scale CLIP feature maps and then compare, this would be a reasonable baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and valuable feedback. Below please find our clarification regarding your comments.\n\n## Reply to weaknesses\n1. **Limitation.** Thank you for  your insightful comment. Indeed our method is trained with a known set of classes, we'd like to add it to the limitations. It's worth mentioning that the baseline Sem(OV-Seg)[3] and Sem(ODISE)[3] also share the same limitation. They require the text labels to first obtain the segmentation maps and they apply the maps for NeRF training. \nTraining with unknown classes and extracting precise object boundaries at inference time is an extremely challenging task for 3D open-vocabulary segmentation on NeRF, which we will explore in future work.\n\n2. **Dataset.** Please refer to the **General Response 1 and 2**.\n\n3. **Major challenge of open-set segmentation.** \nWe would like to respectfully highlight that, in our perspective, the major challenge of **3D** open-vocabulary segmentation is the lack of datasets.  In 2D, we have a large-scale dataset with 400M image-text pairs [1] which is deemed as open-vocabulary. In 3D, we do not have datasets of similar scale and vocabulary. ScanNet200 largely focuses on indoor scenes with common object labels only but lacks a wide range of diverse texts.\n\n4. **Difference with FFD [2].** Please refer to the **General Response 3**.\n\n5. **Proposed baseline.** We adopted a more advanced baseline LERF which can be viewed as your suggested baseline plus additional DINO supervision. In LERF, experiments show that the additional DINO supervision improves the precision of object boundaries clearly. Our method outperforms LERF both quantitatively and qualitatively, hence it should safely outperform your suggested baseline. This is well verified in the new experiments as presented in Table 1 below.\n\nTable 1. Comparisons with proposed baseline. \n| scene | *bench* |  | *sofa* | | *room* | | *table* | |\n|-|-|-|-|-|-|-|-|-|\n|    | **mIoU** | **mAP**   | **mIoU**  | **mAP**   |**mIoU**  | **mAP**   |**mIoU**  | **mAP**   |\n|proposed baseline|50.1|77.6|26.4|42.1|45.6|79.2|32.8|42.2|\n|**ours**         |89.3|96.3|74.0|91.6|92.8|98.9|88.8|96.5|\n\n\n\n## Reply to questions\n\n1. **Selection vector.** \nWe use a NeRF model that has already been optimized through RGB reconstruction. The Selection Volume and feature branch are trained from scratch during the segmentation training stage. Therefore, the selection vector is learned in the segmentation training stage, instead of being obtained from an RGB-optimized NeRF. For more detailed information about the model architecture and training process, please refer to Appendix A.1 and A.3.\n\n2. **Ablations.** We do have the ablation of the 3D selection volume, please see Appendix B for this ablation.\n\nMinor:\n\n1. **Datasets.** We agree with you that collecting 3D open-vocabulary segmentation datasets is a very challenging task as it would require exhaustive labeling of a huge amount of 3D data.\n\n2. **OpenSeg.**  Thank you for the information which will be updated in the Related Work. Nevertheless, we still believe that OpenSeg has smaller vocabulary and knowledge than CLIP as it is trained on a much smaller dataset.\n\n3. **Ambiguous similarities.** Take an image patch with an apple lying on a lawn as an example (lawn is like class A and apple like class B in Fig 2 in the main paper). The corresponding CLIP feature contains both apple and lawn information. If the apple is relatively small, the patch could be classified as lawn as lawn dominates the patch's CLIP features. Thus the class apple would be ignored. Our proposed RDA loss helps address such ambiguity effectively.\n\n4. **Relevancy and segmentation probability distribution.** Segmentation probability means the probability of a pixel belonging to a class. Relecancies mean the similarities between class text features and the images' CLIP features. These terminologies are defined in math in Eq. (5) and Eq. (6) of the main manuscript. We acknowledge the feedback and assure that we will thoroughly refine and highlight the terminology definition in the revised manuscript\n\n[1] Laion-400m: Open dataset of clip-filtered 400\nmillion image-text pairs.\n\n[2] Decomposing NeRF for Editing via Feature Field Distillation\n\n[3] In-Place Scene Labelling and Understanding with Implicit Scene Representation"}, {"review_id": "Fj5deLiZwh", "reviewer": "Reviewer_1xbk", "summary": "This paper introduces a method that incorporates pre-trained CLIP features into Nerf for 3D open-vocabulary segmentation. The authors also utilize pre-trained DINO features to regularize the segmentation process and achieve precise segmentation boundaries. Additionally, they propose a Relevancy-Distribution Alignment loss to address ambiguities in CLIP features.", "questions": "1. How would the performance be affected if the Relevancy-Distribution Alignment was not used during training and instead the segmentation probabilities were spatially normalized during inference?\n2. How does the proposed method perform on Lerf's benchmark?", "limitations": "The processing of multi-spatial features of CLIP incurs substantial computational and storage resource requirements.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper demonstrates a fluent and well-structured organization.\n2. The experimental results exhibit favorable performance.", "weaknesses": "1. The testing data is relatively simple, consisting of views with limited perspectives, and does not include results on larger and more complete 3D scenes (e.g., the benchmark proposed by Lerf).\n2. Regarding the Feature-Distribution Alignment issue: If there are multiple objects of the same class, such as x1, x2, x3, in an image, their DINO features' similarity may be lower than b, but the Feature-Distribution Alignment could make them assigned to different classes, which is unreasonable. Although λ_neg is small, the supplementary material demonstrates the criticality of neg_F, and removing it significantly deteriorates the results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Below please find our clarification regarding your comments.\n\n## Reply to weaknesses\n1. **Experiments on other datasets including LERF dataset.** Please see the **General Response 1 and 2**. We have conducted experiments on more diverse datasets including human scenes, indoor scenes, and a complex scene from LERF datasets.\n\n2. **Scene with multiple instances of a same class.** Please refer to Figure 3 (right) in the rebuttal PDF. We conducted a new experiment to demonstrate that our method can still perform well under such conditions. Since instances of the same class often share similar appearance, texture, etc., they also have similar DINO features. As a result, FDA will not mistakenly segment them into different classes. The RDA loss will further help alleviate this issue by assigning all these instances to the same text label. In the experiment, we observed that our method successfully segments all three apples into the same class with accurate boundaries.\n\n## Reply to questions\n1. **Deferred RDA.** Deferring RDA to the inference stage degrades the performance severely as shown in Table 1 below. Without RDA during training, the trained model yields lots of false label predictions due to the ambiguity of CLIP features, which further degrades FDA in discerning different classes.\n   \nTable 1. Results with deferred RDA.\n| scene | *bench* |  | *sofa* | | *room* | | *table* | | *blue sofa* | |\n|-|-|-|-|-|-|-|-|-|-|-|\n|    | **mIoU** | **mAP**   | **mIoU**  | **mAP**   |**mIoU**  | **mAP**   |**mIoU**  | **mAP**   |**mIoU**  | **mAP**   |\n|deferred RDA|60.7|91.6|66.0|91.3|63.3|94.9|56.6|83.8|14.7|17.6|\n|**ours**    |89.3|96.3|74.0|91.6|92.8|98.9|88.8|96.5|82.8|97.7|\n\n| scene | *covered desk* |  | *bed* | | *lawn* | | *office desk* | | *snacks* | |\n|-|-|-|-|-|-|-|-|-|-|-|\n|    | **mIoU** | **mAP**   | **mIoU**  | **mAP**   |**mIoU**  | **mAP**   |**mIoU**  | **mAP**   |**mIoU**  | **mAP**   |\n|deferred RDA|41.4|58.4|75.6|94.6|70.3|91.3|38.3|55.6|58.3|89.6|\n|**ours**    |88.6|97.2|89.5|96.7|88.2|97.3|91.7|96.2|95.8|99.1|\n\n\n2. **LERF Dataset.** Please see the **General Response 2**."}, {"review_id": "PPVBF6wJq1", "reviewer": "Reviewer_L8we", "summary": "This paper proposes a new approach for fusing in 3D the output of 2D neural networks (CLIP an DINO) applied to the analysis of multiple views of a given static scene. The main application is open-vocabulary 3D segmentation of radiance fields. Compare to trivially fusing CLIP features, the method introduces a few improvements, such as re-normalising the response of CLIP and using DINO to regularise the segments. These improvements are shown to help segmentation significantly. Comparison to recent works, including concurrent work such as LERF, are included.", "questions": "* See the technical issues above. Can you clarify the notation, and, especially, the question about renormalisation in the RDA loss?\n\n* In the scaling experiment of Table 2, bottom 4 rows, you use a very small number of views (according to the text, just 2-3 -- se line 279) are sufficient to obtain good segmentation performance. Do you mean that you are still using all *RGB* views for 3D reconstruction and just 2-3 views with CLIP features, or that you are using only 2-3 features in total, including for 3D reconstruction? The latter would be very surprising.\n", "limitations": "There is no substantive discussion of limitations (e.g., no failure cases demonstrated, no section or paragraph discussing limitations). Note that the NeurIPS guidelines encourage explicitly the discussion of limitations.\n\nThere is no discussion of societal impact, ethics, copyright, data protection etc. I don't believe that any of this is particularly relevant to this paper though. I do not see direct potential harm stemming from this research, and the datasets used appear custom-collected and do not contain personal data.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Fusing open-ended features such as CLIP in 3D can be useful in applications, including, as demonstrated here, for open-vocabulary 3D segmentation.\n\n* I appreciated the effort to compare to very recent *concurrent* work like LERF. The fact that the method outperforms the latter, at least in the authors' experiments, is a further bonus.\n\n* There are a few ideas (class re-balancing, multi-scale feature selection, relevance renormalisation and consistency with DINO features) which are shown to improve significantly the performance compared to simply fusing CLIP features. This illustrates some difficulty with the latter approach, and also shows simple fixes to these issues.\n", "weaknesses": "* The idea of fusing 2D features in 3D reconstruction using neural fields is not particularly novel as such. Semantic NeRF has done it for segmentation, N3F and FFD for unsupervised features like DINO, Panoptic NeRF and others for semantic segmentation, etc. FFD already included some early experiment with open-ended language-based features (and this paper compares to them).\n\n* The math needs significant revision. Some issues amounts to typos and inaccuracies, but others really make it hard to understand how the algorithm works in detail. Some aspects of the algorithm are also slightly surprising and need some discussion. See below:\n   \n     * Eq. (3) is the main loss that allows to learn the radiance field model, including its extension to predict the multi-scale CLIP features F and the scale selection vectors S. I understand most of this *except* why and how this can learn feature selection. First, the equation does not make literal sense. The inner product $\\langle \\hat F(\\mathbf{r}) \\cdot S(\\mathbf{r}) F(\\mathbf{r} \\rangle$ misses a comma, but I expect that the intended read is  $\\langle \\hat F(\\mathbf{r}) \\cdot S(\\mathbf{r}), ~ F(\\mathbf{r} \\rangle$. Second, the $\\cdot$ product is unclear. $F$ is a $N_s \\times D$ matrix and $S$ is a $N_s$-dimensional vector. Is this an element-wise product with broadcast? Thirdly, $\\hat F$ are the multi-scale CLIP features \"fused\" (i.e., rendered) by the radiance field and $F$ the ones predicted from the 2D image. Why comparing these two weighed by $S$ should result in $S$ learning to select the appropriate scale of CLIP features to use for that point?\n\n    * Eq. (6): Is this meant to be $\\langle T, F_I \\rangle$? The notation $\\langle T \\cdot F_I\\rangle$ does not make literal sense. Furthermore, $T$ is a $C\\times D$ matrix and $F_I$ is a $N_s\\times D\\times H\\times W$ matrix. The output is $C\\times H \\times W$, so we can sort of guess what the authors mean to do, but the notation does not show how the scale dimension $N_s$ is reduced (presumably using the scale selector $S$, but the notation does not reflect that).\n\n    * The RDA loss of Eq. (8) is a bit problematic. First, from a modelling perspective, it seems to be based on spatially re-normalizing the response of the CLIP features to all possible classes $c$ an a class-by-class basis (see also Fig. 2). This is done to prevent certain classes from \"swamping\" others. However, what happens if a class is *not* present at all in a scene? Wouldn't this *force* the class to still be segmented, thus incorrectly? Second, there are issues with Eq. (8) as such. The quantity $\\bar R(\\mathbf{r})$ is a $C$-dimensional non-negative vector, but, unless this is done implicitly, it does *not* sum to one. I.e., its is not a *probability* vector. Can you really interrupted Eq. (8) as a sum of JS divergences?\n\n    * Eq. (9)  what is the notation $\\cos f_{hw} \\cdot f_{ij}$? Do you mean $\\langle \\frac{f_{hw}}{\\|f_{hw}\\|},\\frac{f_{ij}}{\\|f_{ij}\\|} \\rangle$  ?\n\n    * What is $z$ in Eq. (9)?\n\n    * I understand that the authors are trying to implement the segmentation distillation idea of [49] -- Eq. (12) certainly bears resemblance to that work. However, I had a really hard time understanding the notation used in Section 3.3 overall.\n\n* The evaluation is only carried out on 10 custom scenes with comparatively little clutter. Given that the authors have been willing to compare to LERF (which is concurrent work, so this is definitely not required) I wonder if it would make sense to run the assessment on *their* 13 custom scenes as well. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and valuable feedback. Below please find our clarification regarding your comments.\n\n## Reply to weaknesses\n\n1. **Novelty.** Please see the **General Response 3**.\n\n2. **Math.**\n\n- In l.162 we use $cos \\langle · \\rangle$ to denote  cosine similarity which could cause confusion. We will modify it to $cos\\langle,\\rangle$. Thus the Eq.(3) becomes:\n$\nL_{supervision} = \\sum_{\\textbf{r} \\in R} \\left(\n\\left \\|  \\hat{C}(\\textbf{r}) - C(\\textbf{r})\\right \\|_2^2 -\n\\cos\\langle\\hat{F}(\\textbf{r}) , S(\\textbf{r})F(\\textbf{r})\\rangle\n\\right).\n$\nThe loss compares the rendered features with the selected features predicted from the 2D image.\nThe Selection Volume is learnt under the supervision of both RDA loss and FDA loss, as the gradients of the two losses are back-propagated to the Selection Volume as well. The loss combination\n\n$\nL_{supervision} + L_{RDA} + L_{FDA}\n$\nallows the Selection Volume to learn the appropriate scale.\n\n- As mentioned before, after modification, Eq. (6) should be:\n$\nR_I = S_I \\cos \\langle T, F_I\\rangle \\in \\mathbb{R}^{C\\times H \\times W}.\n$\nIn Eq.(6) we ignore the spatial dimension $H,W$ for simplicity, which means that during computation, $T$ is broadcast to $C \\times D \\times H \\times W$. Then the cosine similarity $\\cos \\langle T, F_I\\rangle$ is performed on the first two dimensions, making $\\cos \\langle T, F_I\\rangle$ a $C \\times N_s \\times H \\times W$ matrix. Then $S_I$ is reshaped as $N_s \\times 1 \\times H \\times W$. Then the $R_I$ is the dot product between $\\cos \\langle T, F_I\\rangle$ and $S_I$ in the first two dimensions. The dot product is $C \\times 1 \\times H \\times W$ which is then reshaped to $C\\times H \\times W$. **Because there are lots of details which can be better explained with code, we thus use such a simple notion here** in Eq.(6) to give readers a basic idea about how our algorithm works. We'll include the details in the appendix and release the code later. \n\n- First, our method is trained with a known set of classes in the scene, we'd like to add it to the limitations. It's worth mentioning that the baseline Sem(OV-Seg)[1] and Sem(ODISE)[1] also share this same limitation. They require the text labels before the training stage to obtain the segmentation maps used for training the NeRF. Second, before computing the Eq.(8), we apply a softmax function to $\\bar{R}(\\textbf{r})$ to make it a probability vector. We skipped this detail due to the space limit and we will include it in the appendix later.\n\n- As mentioned before, after modification, Eq.(9) becomes\n$\n\\cos \\langle f_{hw}, f_{ij} \\rangle\n$.\n\n- $z$ is segmentation logits defined in Eq.(4).\n\n3. **More experiments.** Please refer to the **General Response 1 and 2**. We have conducted experiments on more diverse datasets including human scenes, indoor scenes, and complex scenes.\n\n## Reply to questions\n1. Please see the reply to the weaknesses.\n\n2. We use all views for 3D reconstruction and just 2-3 views with CLIP features in the studies with 10% views. \n\n[1] In-Place Scene Labelling and Understanding with Implicit Scene Representation"}]

Author rebuttal:
We are sincerely thankful to every reviewer for reading our research narrowly and giving us thoughtful feedback. We carefully respond to the main concerns raised by the reviewers as below.

## 1 Evaluation Datasets

Reviewers **t9cD** and **kCD4** shared that our collected dataset has a limited scale for evaluating open-vocabulary features. We would clarify that 3D open-vocabulary segmentation is a relatively new task with few relevant benchmarks available to the best of our knowledge (so the very recent and related LERF [6] also evaluates on its own collected dataset). Specifically, most existing 3D datasets are either unannotated or have very limited object/scene classes. To tackle this issue, we purposely collected images of a wide range of diverse objects such as electronic products, fruits, animals, human body parts, brands, instruments, food, plants, etc., aiming to offer a new baseline for benchmarking on 3D open-vocabulary segmentation. In fact, our dataset is much more challenging than those used in the recent NeRF-based segmentation methods [1, 2] (from NeurIPS 2022 and ICLR 2023), from both object number per scene and complexity of scene geometry. Nevertheless, we managed to conduct more evaluations on a few other datasets as described below.


## 2 Evaluations on Other Datasets and Settings

We additionally perform evaluations on the human body dataset, human head dataset, indoor datasets with low-quality images [4, 5], and a complex scene from LERF datasets [6]. We compare with the ***concurrent*** work LERF [6] qualitatively due to the lack of labels or the defective annotations as pointed out in [3]. We also perform experiments with different text prompts. All the newly conducted qualitative experiments are presented in the submitted rebuttal PDF.

**Human body and head.** As shown in Figure 1 in the rebuttal PDF, our method segments more precise parts than LERF. Specifically, LERF fails to segment the "head" in the human body and the "black T-shirt" in the human head. In contrast, our method can recognize and segment these parts correctly because our designed RDA loss addresses the ambiguity of the CLIP features effectively.

**Indoor scenes with low-quality images.** Figure 2 in the rebuttal PDF shows experiments on the indoor datasets [4, 5], where many images are unrealistically rendered with less photorealistic appearances (as indicated in [3]) and have limited spatial resolution (640 $\times$ 480 or 1024 $\times$ 768). Due to these data constraints, our method sometimes confuses labels with similar appearances. However, we can see that our method still outperforms LERF by a large margin by successfully segmenting more labels.

**Complex scene.** Figure 3 (left) in the rebuttal PDF shows the segmentation of one challenging sample from the LERF dataset [6], where the scene has complex geometry as well as many objects of varying sizes. It can be observed that LERF cannot segment most objects due to the ambiguities of CLIP features while our method can segment more objects correctly with more precise boundaries.

**Segmentation with different text prompts.**  Reviewer **1C3c** raised questions about the impact of the text prompts, thus we conduct experiments to segment scenes with different text prompts.  In the experiments, we replaced the original texts with different languages (e.g., Portuguese egg tart -> Pastel de Nata),  names (e.g., dressing doll -> Barbie), and actions (e.g., hand soap -> wash hand, black headphone -> listen to music). As Figure 4 in the rebuttal PDF shows, with the rephrased text prompts, our method can still segment the scenes reliably. The experiments are well aligned with the quantitative experiments as shown in Table 1 below. 

Table 1. Results with different text prompts.
| scene | *lawn* |  | *bench* | |
|-|-|-|-|-|
|    | **mIoU** | **mAP**   | **mIoU**  | **mAP**   |
| original   | 88.2   | 97.3   | 89.3   | 96.3  |
| rephrased  | 89.3  | 97.2  | 88.4  | 96.6  |


## 3 Novelty 

Reviewers **kCD4** and **L8we** expressed concerns about the novelty of our approach, particularly when compared to FFD [6]. We would like to emphasize that our contribution is significantly different from FFD and other prior studies in three major aspects:

1. Our method directly distills open-vocabulary knowledge from CLIP, instead of relying on a fine-tuned version like LSeg. This enables the exploitation of the original large-scale vocabulary in CLIP.

2. Our method exploits two complementary foundation models, including DINO for precise object boundaries and CLIP for open vocabulary. In contrast, FFD and other methods like N3F [7] use a single foundation model at a time.

3. Our approach introduces a novel pipeline for 3D open-vocabulary segmentation using NeRF and achieves better performance than previous methods without requiring any segmentation annotations. Specifically, we introduce the Selection Volume to generate pixel-aligned CLIP features, RDA to mitigate the ambiguities of the CLIP features, and FDA for precise object boundaries.

**We firmly believe that our method can significantly benefit the 3D vision community by showcasing the possibility of learning 3D open-vocabulary segmentation from unlabeled 2D images and text-image pairs.**



[1] NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes

[2] Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation

[3] Decomposing NeRF for Editing via Feature Field Distillation

[4] Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

[5] The Replica Dataset: A Digital Replica of Indoor Spaces

[6] LERF: Language Embedded Radiance Fields

[7] Neural Feature Fusion Fields (N3F): 3D Distillation of Self-Supervised 2D Image Representations

Meta-review: This paper proposes to train a NeRF model by leveraging two self-supervised trained models CLIP and DINO, tackling the problem of 3D scene segmentation. It receives two weak accepts, one borderline accept, one borderline reject and one reject.
During discussion, Reviewer kCD4 still got concerns on the comparison with LERF and the experiments for"Open-Vocabulary". 
Reviewers had in-depth discussion and the AC recommends acceptance for this paper.
The authors are urged to clarify reviewers' concerns in the revision.
Decision: Accept (poster)