OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views
Authors: Francis Engelmann
Fabian Manhardt
Michael Niemeyer
Keisuke Tateno
Federico Tombari
Venue: ICLR 2024 poster (2024)
Keywords: 3D
open set segmentation
neural radiance fields
VLM features
Primary area: applications to robotics, autonomy, planning
TL;DR: OpenNeRF outperforms recent open-vocabulary methods for open set segmentation by using a neural representation that allows for incorporating pixel-wise VLM features and rendering novel views.

Abstract: Large visual-language models (VLMs), like CLIP, enable open-set image segmentation to segment arbitrary concepts from an image in a zero-shot manner. This goes beyond the traditional closed-set assumption, i.e., where models can only segment classes from a pre-defined training set. More recently, first works on open-set segmentation in 3D scenes have appeared in the literature. These methods are heavily influenced by closed-set 3D convolutional approaches that process point clouds or polygon meshes. However, these 3D scene representations do not align well with the image-based nature of the visual-language models. Indeed, point cloud and 3D meshes typically have a lower resolution than images and the reconstructed 3D scene geometry might not project well to the underlying 2D image sequences used to compute pixel-aligned CLIP features. To address these challenges, we propose OpenNeRF which naturally operates on posed images and directly encodes the VLM features within the NeRF. This is similar in spirit to LERF, however our work shows that using pixel-wise VLM features (instead of global CLIP features) results in an overall less complex architecture without the need for additional DINO regularization. Our OpenNeRF further leverages NeRF’s ability to render novel views and extract open-set VLM features from areas that are not well observed in the initial posed images. For 3D point cloud segmentation on the Replica dataset, OpenNeRF outperforms recent open-vocabulary methods such as LERF and OpenScene by at least +4.9 mIoU.

Published as a conference paper at ICLR 2024
OPENNERF: OPEN SET 3D NEURAL SCENE SEGMEN- TATION WITH PIXEL- FEATURES AND RENDERED NOVEL VIEWS
Francis Engelmann 1.2, Fabian Manhardt', Michael Niemeyer", Keisuke Tateno", Marc Pollefeys Federico Tombari2.4
EET Zurich, Google, Miirossoft, 'TU Munich
ABSTRACT
Large visual-language models (VLMs), like CLIP, enable open-set image seg- mentation to segment arbitrary concepts from an image in zero-shot manner. This goes beyond the traditional closed-set assumption, i.e., where models can only segment classes from pre-defined training set. More recently, first works on open-set segmentation in 3D scenes have appeared in the literature. These methods are heavily influenced by closed-sct 3D convolutional approaches that process point clouds or polygon meshes. However, these 3D scene representa- tions do not align well with the image-based nature of the visual-language mod- els. Indeed, point cloud and 3D meshes typically have a lower resolution than images and the reconstructed 3D scene geometry might not project well the underlying 2D image sequences used to compute pixel-aligned CLIP features. To address these challenges,
propose OpenNeRF which naturally operates posed images and directly encodes the VLM features within the NeRF. This is similar in spirit to LERF, however our work shows that using pixel-wise VLM features (instead of global CLIP features) results in overall less complex ar- chitecture without the need for additional DINO regularization. Our OpenNeRF further leverages NeRF's ability render novel views and extract open-set VLM features from areas that are not well observed in the initial posed images. For 3D point cloud segmentation the Replica dataset, OpenNeRF outperforms recent opea-vocabulary methods such as LERF and OpenScene by at least +4.9 mloU.
INTRODUCTION
3D semantic segmentation of a scene is the task of estimating, for each 3D point scene, the category that it belongs to. Being able accurately estimate the scene semanties enables numerous applications, including AR/VR, robotic perception (Zhi et al., 2021; Zurbrügg et al., 2024) and autonomous driving (Kundu al., 2022; Kreuzberg et al., 2022), since they all require having fine-grained understanding of the environment. While the domain of 3D scene understanding has recently made lot of progress (Schult et al., 2023; Choy al., 2019; Yue et al., 2023; Qi et al., 2017; Takmaz et al., 2023b), these methods are exclusively trained in fully supervised manner on (closed-)sed Jsets of semantic categories, rendering them impractical for many real world applications
Inpul Scene
LERF
OpenScene
OpenNeRF (Ours)
Wall Celling @ Flor Blinds Window Lamp Docr Pilow Cohirer Bianiett Cashioa Bed @ Picure IndocePlant @ Sher Vise wall Plug Book Swach
Figure 1: Opca-vocabulary 3D semantic segmentation on point clouds. Compared to LERF (Kerr et al., 2023), accurate and better localized, while achieving more finc-
the segmentation masks of OpenNeRF grained classification than OpenScene (Peng et al., 2023). Zero-shot results on Replica (Straub et al., 2019).

---

Published as a conference paper at ICLR 2024
"pillow on sofa"
We propose OpenNeRF, an approach for open-set 3D seene understanding based neural radiance fields. Arbitrary concepts he queried from our representation (left). As the original camera trajectory (blue, middle) might not capture all interesting scene details, we use NeRFs ability to render novel views (right) and propose a mechanism to obtain relevant novel camera poses (yellow, middle) that focus on scene details from which we can extract additional open-scene features improving the overall open-set scene representation.
Figure 2:
as the models lack the flexibility to continuously adapt to novel concepts (Delitzas et al., 2024) or semantic classes (Dai al., 2017a; Armeni et al., 2017; Ramakrishnan et al., 2021; Baruch et al., 2021: Straub et al., 2019). Therefore, in this work we aim at tackling the problem of open-set 3D scene segmentation. The main idea of open-set scene segmentation that arbitrary concepts can be segmented, independent of any pre-defined closed set of classes. Given an arbitrary query - for example, textual description or an image of an object - the goal to segment those parts in the 3D scene that are described by the query. Such general unconstrained functionality can be crucial for helping robots interact with previously unseen environments, or applications on AR/VR devices in complex indoor scenes, especially when training labels are scarce unavailable (Ahn al., 2022).
Vision-language models (VLMs), such as CLIP (Wang et al., 2022) or ALIGN (Jia et al., 2021), have shown impressive performance open-set image classification. Trained interet-scale image- caption pairs, they learn a joint embedding mapping text and image inputs to the same (or different) embedding vector depending whether they describe similar (or different) concepts. More cently, these powerful concepts were applied to dense pixel-level tasks, enabling open-set 2D image segmentation (Ghiasi al., 2022; Li et al., 2022a; Rao al., 2022). Using VLMs in combina- tion with 3D point clouds is to date less explored. Similar to Li et al. (2022a), Rozenberszki et al. (2022) train 3D convolutional network that predicts per-point CLIP features in a fully-supervised manner using the CLIP-text encodings of the annotated class names. Such fine-tuning on densely annotated datasets works well only on the classes labeled in the training set but does not generalize well to novel unseen classes which limits open-set scene understanding. OpenScene (Peng et al., 2023) recently proposed the first exciting results on ppen-vocabllary 3D scene understanding. In line with Rozenberszki et al. (2022), given reconstructed 3D point cloud as input, 3D convolu- tional network predicts CLIP features for each point. However, unlike Rozenberszki et al. (2022), the model is supervised with projected CLIP- features from posed 2D images, preserving the generalization ability of the pixel-aligned visual- linguale image features. Similarly, LERF (Kerr al., 2023) embeds CLIP features at multiple scales into neural radiance fields. As in Kobayashi et al. (2022), LERF adds branch to predict CLIP features for given 3D location. Subsequently, this enables rendering CLIP images, which is used to perform open-set semantic segmentation by means of computing the similarity with the CLIP feature from encoded input text queries.
While OpenScene (Peng et al., 2023), OpenMask3D (Takmaz et al., 2023a), and LERF (Kerr al., 2023) demonstrate impressive capability of segmenting any given concept, they still suffer from several limitations. Exemplary. OpenScene fully operates on a given 3D scene reconstruction in the form of a polygon mesh (or a point cloud sampled from the mesh surface) that usually stems from a recorded RGB-D sequence. This approach naturally limited by the mesh resolution, thus constraining the representation of smaller objects. On the other hand, LERF does not exhibit this limitation. However, as relies on the CLIP-image encoder, the resulting global VLM features are not well localized in 3D space which leads to segmentation masks that a fairly inaccurate as CLIP can only be computed on full images or crops. To compensate for this issue, LERF employs multi- resolution patches for CLIP computation and further regularizes the rendered CLIP embeddings using DINO features. While this can mitigate inaccurate masks to some extent, it increases overall model complexity, yet achieving inferior open-set 3D segmentation masks (see Fig. 1).
2

---

Published as a conference paper at ICLR 2024
In this work, we address the aforementioned limitations and propose OpenNeRF, novel neural radi- ance field (NeRF) (Mildenhall et al., 2020) based approach for open-set 3D scene segmentation. As neural representation, NeRFs have inherently unlimited resolution and, more importantly, they also provide an intuitive mechanism for rendering novel views from arbitrary camera positions. Thus, we leverage this ability to extract additional visual-language features from novel views leading to im- proved segmentation performance. One key challenge is to determine the relevant parts of the scene requiring further attention. We identify the disagreement from multiple views as powerful signal and propose probabilistic approach to generate novel view points. Second, we further propose the direct distillation of pixel-aligned CLIP features from OpenSeg (Ghiasi et al., 2022) into our neural radiance field. Compared to LERF, this not only increases the segmentation quality via de- tailed and pixel-aligned CL.IP features, it also significantly simplifies the underlying architecture as it eradicates the need for multi-resolution patches and additional DINO-based regularization terms.
Our experiments show that NeRF-based representations, such as LERF and ourOpenNeRRF are bet- ter suited for detecting small long-tail objects compared to mesh based representations used e.g., in OpenScene. Further, we find that by incorporating pixel-aligned CLIP features from novel views, system improves over prior works despite exhibiting simpler design. We identify the Replica dataset as a promising candidate to evaluate open-set 3D semantic segmentation since, unlike Scan- Net (Dai al., 2017a) Matterport (Ramakrishnan al., 2021), it comes with very accurate mesh reconstruction, per-point semantic labels as well as a long-tail class distribution (see Fig. 4).
In summary, the contributions of this work are as follows:
We propose OpenNeRF, novel approach for open-set 3D semantic scene understanding based on distillation of pixel-aligned CLIP features into Neural Radiance Fields (NeRF).
We propose a mechanism utilizing NeRF's view synthesis capabilities for extracting addi- tional visual-language features, leading to improved segmentation performance.
We present the first evaluation protocol for the task of open-set 3D semantic segmentation, comparing explicit-based (mesh point cloud) and implicit-based (NeRF) methods.
OpenNeRF significantly outperforms the current state-of sate-of-the of-the-art for open-vocabulay 3D segmentation with an +4.5 mloU gain on the Replica dataset.
2
RELATED WORK
Features. CLIP (Wang et al., 2022) is large-scale visual- lisual-anguage model trained on Internet-scale image-caption pairs. It consists of an image-encoder and text-encoder that map the respective inputs into shared embedding space. Both encoders trained in contrastive manner, such that they map images and captions to the same location in the embedding space if the caption describes the image, and different locations in the opposite case. While the CLIP image- encoder yields a single global feature vector per image, LSeg (Li et al., 2022a) extends this idea and predicts pixel-level features which enables dense image segmentation tasks. Pixel-aligned CLIP features are obtained via fine-tuning on fully-annotated 2D semantic segmentation dataset. This works well for the semantic classes present in the fully-annotated dataset, however, the pixel-aligned features generalize less well to novel concepts outside of the training classes. OpenSeg (Ghiasi et al., 2022) further improves on these aspects and proposes a class-agnostic fine-tuning to obtain pixel-aligned features. OVSeg (Liang et al., 2023) is the latest development that proposes to fine- tune a CLIP-like model on cropped objects. In this work, we use the pixel-wise features from OpenSeg (Ghiasi et al., 2022) which enables direct and fair comparison with the publicly available models of OpenScene (Peng et al., 2023). Further, as also noted by (Peng et al., 2023), OpenSeg improves over LSeg (Li et al., 2022a) on long-tail classes unseen during the training of LSeg.
2D
Neural Radiance Fields. Since the introduction of Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) for view synthesis, they have been adopted as scene representation for various tasks ranging from 3D reconstruction (Oechsle et al., 2021; Wang et al., 2021; Yu et al., 2022; Yariv et al., 2021) to semantic segmentation (Zhi et al., 2021) due their simplicity and state-of-the-ar performance. Next to impressive view synthesis results, NeRFs also offer flexible way of fus- ing ba-bd information in 3D. A series of works have explored this property in the context of 3D semantic scene understanding. In Panopticl.itting (Siddiqui et al., 2023), predicted 2D seman-
3

---

Published as a conference paper at ICLR 2024
tie maps utilized to obtain 3D semantic and instance-segne segmented representation of the scene. In (Kobayashi et al., 2022), 2D semantic features are incorporated and fused during the NeRF opti- mization which are shown to allow for localized edits for input text prompts. Neural Feature Fusion Fields (Tschernezki et al., 2022) investigates the NeRF-based fusion of 2D semantic features in the context of 3D distillation and shows superior performance to 2D distillation. Finally, in LERF (Kerr et al., 2023), NeRF-based 3D CLIP-feature fields are optimized via multi-scale 2D supervision to obtain scene representations that allow for rendering response maps for long-tail open-vocabulary queries. While all of the above achieve impressive 3D fusion results, we propose to investigate NeRF-based feature fusing in the context of open-set 3D scene segmentation. This does not only require to solve additional challenges, such as detecting relevant parts of the scene, but also enables more rigorous evaluation and comparison to other fusion approaches.
3D Open-Set Scene Understanding While most approaches for 3D scene understanding utilize 3D supervision (Han et al., 2020; Hu et al., 2021; Li al., 2022b), a recent line of works investi- gates how 2l)-based information can be lifted to 3D. In Semantic Abstraction (Ha & Song, 2022), 2D>baaedd CLIP features are projected 3D space via relevancy maps, which are extracted from an input RGB-D stream. While achieving promising results, their 3D reasoning is coarse and thus limited. Similarly, in ConceptFusson (Jatavallabhula et al., 2023) multi-modal 3D semantic repre- sentations are inferred from an input RGB-D stream by leveraging 2D foundation models. They use point clouds as 3D representation. ScanNer200 (Rozenberszki et al., 2022) uses CLIP to investigate and develop novel 200-class 3D semantic segmentation benchmark. Most similar to our approach is OpenScene (Peng et al., 2023), which the only existing method for open-set 3D semantic seg- mentation of point clouds. Our method is also similar to LERF (Kerr et al., 2023) in terms of NeRF representation but does not demonstrate results on 3D semantic segmentation. Further, our NeRF- based scene representation offers high-quality novel view synthesis which we leverage to render novel views of "interesting" scene parts towards improved segmentation performance.
METHOD
Given a set of posed RGB input images and corresponding pixel-aligned open-set image features, we want to obtain a continuous volumetric 3D scene representation that can be queried for arbitrary concepts. Our approach OpenNeRF enables open-set 3D scene understanding to address a wide variety of different tasks, such material and property understanding as well as object localization. By means of leveraging the normalized cosine similarity between the encoded queries and the ren- dered open-set features, we can explore multiple concepts (see Fig. 6). Alternatively, can by seen as a method for unsupervised zero-shot 3D semantic segmentation (Fig. 5) where we assign each point the semantic class with the highest similarity score.
approach
3.1 OPEN-SET RADIANCE FIELDS
radiance field is a continuous mapping that predicts a volume density a E [0. oo) and a RGB color [0, 1] for given input 3D point E RR3 and viewing direction d $2. Mildenhall (Mildenhall et al., 2020) propose to parameterize this function with a neural network (NeRF) using multi-layer perceptron (MLP) where the weights of this MLP are optimized to fit set of input images of a scene. To enable higher-frequenge modeling, the input 3D point as well as the viewing direction are first passed to a predefined positional encoding (Mildenhall et al., 2020: Tancik al., 2020) before feeding them into the network. Building on this representation, we additionally assign an open-set feature o E R to each 3D point:
al.
fx(x, - (x (o,
(1)
where 0 indicates the trainable network weights. Our representation is based on Mip-NeRF (Barron et al., 2022) for appearance and density, while an additional MLP head models the open-set field. Es- sentially, the rendering of color, density and the open-set features is conducted following volumetric rendering via integrating over sampled point positions along a ray (Levoy, 1990). We supervise the open-set head with OpenSeg feature maps which provide localized per- per-pixel CLIP features. As sult, we do not require rendering CLIP features at multiple scales for training or multiple renderings for each scale during inference in LERF, leading simpler and more efficient representation.
4

---

Published as a conference paper at ICLR 2024
3.2 TRAINING OBJECTIVES
Appearance Loss. Following standard procedure, we optimize the appearance using the Euclidean distance between the rendered color e and the ground truth color e, over of sampled rays R:
CRGB
1 IR
Σ |lc, c,
(2)
PE
Depth Loss. While our method is able to be trained from RGB data alone, we can also leverage depth data if available. To this end, supervise the density for those pixels where observed depth information is available (see Table 2 for an analysis). We compute the mean depth from the densities for each ray and supervise them using the smooth L1 (Huber) loss:
daeph
Σ FER
l|d, d,
(3)
R
Open-Set Loss. The open-set set features are supervised via pre-computed 2D open- set feature maps from OpenSeg. Similar to Peng et al. (2023), we maximize the cosine similarity using
oopen
(4)
R
1,
1
PER
Since the 2D open-set feature maps are generally not multi-view consistent, we do not backpropagate
from the open-set feature branch to the density branch (Siddiqui al., 2023; Kobayashi al., 2022). Additionally, the 2D open-set maps from OpenSeg (Ghiasi et al., 2022) contain many artifacts near the image border. Therefore, we do not sample rays within a margin of 10 pixels from the image border during training. In summary, the total loss is defined as f LRGB
Aupen' Logen + Adepth Lelepth
3.3 RENDERING NOVEL VIEWS
key advantage of NeRF-based representations is their ability to render photo-realistic novel views. These rendered novel views can naturally be used to extract 2D open-set features. We would like to use this ability to obtain improved open-set features for those parts of the scene where we have low confidence in the existing open-set features. A key challenge, however, is to first identify the parts of the scene that exhibit low confidence features and would thus benefit from rendering novel views.
Confidence Estimation. We identify the uncertainty 14 R over multiple projected open-set features as a surprisingly strong signal. Specifically, starting with the original color images, we compute open-set feature maps with OpenSeg. We then project each feature map onto coarse 3D scene point cloud obtained using an of-the-shelf 3D reconstruction method (Dai et al., 201 and compute for each point i the mean 14 TR. and covariance E, RDDD over the per-point projected features. As per-point uncertainty measure 1932) defined u, det( Those parts of the scene that exhibit a large uncertainty intuitively correspond to areas where the open-set features from multiple viewpoints disagree on. They hence deserve further investigation by means of re-rendering from more suitable viewpoints. In practice, calculating the variance over large number of nigh-dimensional features can be computational challenging. For a memory efficient and numerically stable implementation, we rely on Welford's online algorithm (Donald et al., 1999) to compute the variance in single pass.
E E we compute the generalized variance (Wilks,
To confirm that idea, we compute the correlation between the per-point uncertainty u, and the per- point error e;. We define the per-point error e; the Euclidean distance between the ground truth open-set vector of (i.e., the CLIP text-encoding of the annotated class name) and the per-point mean open-set vector [4;, Indeed, measured over all Replica scenes, we observe strong positive correlation (F = 0.653) between i, and e,. See Fig. 3 for an illustration.
Novel Camera View Selection. To generate novel camera poses we compute the lookat matrix based a target t E R3 and camera position p E R² As targets t we select points with high uncertainty (shown in red, Fig.3) based on their uncertainty u, if x < u, with x ~ Uniform 1]. fixed small offset from the target position inside the scene with
The camera position is placed added small random noise. Importantly, we sample the density of the NeRF at the selected camera position to make the target is visible and does not collide with the scene geometry.
5

---

Published as a conference paper at ICLR 2024
crror E3
r 0.69
eirrr 6;
uncertainly
uncertainty u;
Figure 3: Confidence Estimation. The enror (left) correlates well with the estimated uncertainty (center). Our mechanism for selecting novel view points is based the estimated uncertainty. The plot (right) shows the error-uncetaannty correlation r for room0 of the Replica (Straub et al., 2019) dataset.
3.4 IMPLEMENTATION AND TRAINING DETAILS
Our model is implement in Jax based on Mip-NeRF (Barron et al., 2022). We train each NeRF scene representation for 3000 iterations and the pixel-aligned opens-set set features computed using OpenSeg (Ghiasi et al., 2022) resulting in 640 x 360 x D dimensional feature maps with D = 768. For memory efficiency reasons, convert them f values. For querying, we use the pre- trained CLIP text-encoder based on the ViT-144 @1:144@336 336 model (Wang et al., 2022).
EXPERIMENTS
Datasets. We evaluate our approach on the Replica (Straub et al., 2019) dataset and show qualita- tive results on scenes captured with the iPhone 3D Scanner App. Replica consists of high quality 3D reconstructions of variety of real-world indoor spaces with photo-realistic textures. Unlike other popular 3D semantic segmentation datasets, such as S3DIS (Armeni et al., 2017), Scannet (Dai et al., 2017a) Matterport (Ramakrishnan et al., 2021), Replica is particularly well suited to eval- uate open-set 3D scene understanding as contains both a long-tail class distribution and carefully- annotated ground-truth semantic labels, including very small objects such as switches and wall- plugs. All experiments are evaluated on the commonly-used 8 scenes {offi ce 0-04,,
[pom0-2] (Zhu al., 2022: Peng et al., 2023; Zhi al., 2021). using the camera poses and RGB-D frame se- quences from Tice-SLAM (Zhu et al., 2022). Each RGB-D video sequence consists of 2000 frames scaled to 640 x 360 pixels. For a fair comparison OpenScene (Peng et al., 2023), we use only every 10-th frame for training resulting in 200 posed RGB-D frames per scene.
Labels and Metrics. Overall, the 3D scene reconstructions are annotated with 51 different se- mantic class categories. To enable more detailed analysis of the experiments, we further split the original categories into three equally sized subsets (head, common, tail) based on number of annotated points where each subset contains 17 classes (see Fig. 4). Note that the ground-truth semantic labels, however, are only used for evaluation and not for optimizing the representations.
Log number points per
15-
Due
now
do
timp
pat
houl sio Antoo
Figure 4: Class Frequency Distribution of the Replica Dataset (Straub et al., 2019). We show the number of point annotations for each category. The colors indicate separation head (blue), common (yellow) and tail (green) classes from left to right in decreasing order. Note that the plot shown log-scale.

---

Published as a conference paper at ICLR 2024
All
Head
Common
Tail
mloU
mAce
mloU
mAcc
mloU
mAcc
mloU
mAcc
LERF (Kerr al., 2023)
10.5
25.8
19.2
28.1
10.1
31.2
2.3
17.3
OpenScene (Peng et al., 2023) (Distilled)
14.8
23.0
30.2
41.1
12.8
21.3
1.4
6.7
OpenScene (Peng et al., 2023) (Ensemble)
15.9
24.6
31.7
44.8
14.5
22.6
1.5
6.3
OpenNeRF (Ours)
20.4
31.7
35.4
46.2
20.1
31.3
5.8
17.6
Table 1: 3D Semantic Segmentation Scores on Replica (Straub et al., 2019). All results are obtained from image resolution of 640 360 pixels, using the OpenSeg image encoder and the CLIP encoder hased on ViT- 336 and averaged over three runs. The reported LERF results are obtained using their original implementation adapted for Replica following the same experimental setup as ours. OpenScene scores are obtained with their provided pre-trained models. Note that the OpenScene models additionally profit from pre- traiiigg the large-scale Matterport (Ramakrishnan et al., 2021) dataset.
For evaluation purposes, we compute the 3D semantic segmentation performance on the provided 3D scene point clouds. In particular, follow (Peng et al., 2023) and measure the accuracy of the predicted semantic labels using the mean intersection over union (mloU) and mean accuracy (mAcc) over all ground truth semantic classes and the head, common, tail subsets.
4.1 METHODS IN COMPARISON.
We compare our approach to the recently proposed OpenScene (Peng et al., 2023) and LERF (Kerr al., 2023). OpenScene is currently the only method reporting open-world 3D semantic segmenta- tion scores. For LERF, we obtain 3D semantic segmentation masks by rendering for each frame all relevancy maps over all evaluated semantic classes, then project these on the Replica point clouds and assign the semantic class with the highest summed relevancy score. The OpenScene model is a sparse 3D convolutional network that consumes a 3D point cloud and predicts for each point an open-set feature. The model is trained on large-scale 3D point cloud datasets and supervised with multi-view fused CLIP features. We compare with the two variations of their trained model, the 3D distilled model (Distilled), which directly predicts per-point features, and the improved 2D-3D en- semble model (Ensemble), which additionally combines the predicted per-point features with fused pixel features. For fair comparison, we follow their experimental setup, using the same pixel- aligned visual-language feature extractor OpenSeg (Ghiasi et al., 2022). We use the public code and the Matterport (Ramakrishnan et al., 2021) pre-trained model suggested in the official repository Note that OpenScene profits from additional training datasets not used by LERF or our OpenNeRF.
4.2 RESULTS ON 3D SEMANTIC SEGMENTATION
We show 3D semantic segmentation scores for all three subsets in Table 1. The 3D scene segmen- tations are obtained by querying the 3D scene representations for each one of the annotated ground truth semantic classes and assigning the class with the highest similarity score each 3D point. Querying is performed via correlation of the 3D point cloud features with the embedding from large language models, specifically the CLIP-text encoder VIT-L.14@336) For OpenScene, we observe a similar trend as reported in (Peng et al., 2023), where the Ensemble model improves over the Distilled model (+1.1 mloU).
Our results clearly improve over all baseline methods. We outperform LERF significantly while possessing a simpler overall design. Compared to OpenScene, we achieve +4.5 mloU over all classes, -.77 (head), +5.6 (common) and +4.3 (tail) for each subset. While achieving improved results on all classes, the smallest improvement can be found for the head classes. We attribute this to the fact that OpenScene can benefit from the pre-training on large-scale 3D datasets in this setting, enabling OpenScene to learn geometric priors of more popular classes (wall, ceiling, floor, chair). However, that are not recognized at all by OpenScene (wall-plug, clock, tissue-paper, tablet, cloth). This is an important aspect, as these are often exactly the classes that are most relevant for an autonomous agent to interact with. Nevertheless, we also observe that numerous toil- classes are not detected all by neither method. This highlights that ppen-scene segmentation is a difficult task especially for long-tail classes, and shows that Replica is challenging dataset for benchmarking.
interesting to note that our approach is able detect semantic categories
hhtpp:://gii http:///ggggtuub.c.oeeegggnnnnne htwpbb/www/gggggggggo///ee..ecmm
7

---

Published as a conference paper at ICLR 2024
Color
Ground Trush
Calor
Ground Truth
OpenSoene
OpenScone
Wall @ Ceiling @ Floor @ Chair Blinds Sofa Tuble Rug @ Window Lampp Door @ Pillow Bench @ TV Screen @ Cabinet Pillar Blanket
TV Stand & Cishion Rin Van Rail @ Steel Pizture IndoorPant Desk Conforter Nighistan Saclf Vow PlanStund Bookes @ Platc Monitor
Pipe Pandi Dest Organizer Wall Plug Book @ Box Clock Sculptzee
Tissa: Paper Canera
Tables Pot Bonll @ Cundic Bood Clath Switch
Figure 5: Qualitative 3D Segmentation Results and Comparison with OpenScene (Peng et al., 2023). The white dashed circles indicate the most noticcable differences between both approaches. Color and ground truth are shown for reference only. Overall, our approach produces less noisy segmentation masks.
Toower pot"
"pillow top coucc""
"swirch"
light switchess
Objjects
painting
"urtissti"
mirror
"reflective"
"way""
bed sheets
Properties
"wood""
"slaxz"
Materials
Figure 6: Open-Set Scene Exploration. We visualize the normalized cosine similarity between the rendered open-set scene features and the encoded text queries shown the top-right of each examplc. Examples cover broad range of concepts. Going beyond specific objects (lop), we show scene properties (middle) and various materials (bottom). Red is the highest relevancy, green is middle, blue the lowest. Uncolored means the similarity values are under 0.5.
'moot"
"musooi"
Sou"
Figure 7: In-the-wild Qualitative Results. We capture scenes using the iPhone 3D Scunner App, which provides camera poses from on-board SLAM and query the reconstructed scenes with various search terms.
8

---

Published as a conference paper at ICLR 2024
4.3 ANALYSIS EXPERIMENTS
Sampling or rendering? Unlike OpenScene, which directly predicts per-point features for each point of given 3D point cloud, our NeRF representation more versatile. We can either directly sample ! the open-set features at a specified 3D position from the NeRF represen- tation, or we can first render and then project ( the open-set features onto the 3D point cloud for evaluation. When multiple open-set fea- tures are projected to the same 3D point we take the average over all points. Note that in both cases the 3D point cloud is only required for evaluation and, in contrast to OpenScene, not necessary to obtain the NeRF-based scene representation, which only relies on posed RGB(-D) images. Table 2 shows that the projection approach @ improves over sampling ①. which can be a direct consequence of the volumetric rendering within NeRFs that accumulates multiple samples along each ray compared that L already improves over both OpenScene and LERF.
mloU
mAcc
LERF (Kerr al., 2023)
10.4
25.5
OpenScene (Peng et al., 2023)
15.1
24.6
Sampled
16.5
29.1
Render & Project
18.5
29.8
depth supervision
19.4
30.8
rendered novel views
20.4
31.7
Table 2: Analysis Experiments.
single sample at a given 3D point. Note
Impact of Depth Supervision. We further analyze the importance of depth as additional supervi- sion signal. We implement an additional regression loss using the Huber (smooth-L4) loss between the rendered average distance and the ground truth depth from each training pose. Table 2, A shows that depth supervision further improves the open-set feature field since the additional depth supervision has direct impact on the volumetric reconstruction quality.
Impact of Novel Views. Next, we analyze the contribution of the rendered novel views from the generated view points using the approach described in Section 3.3. Table 2, ① clearly demonstrates the increased performance from rendered novel views. To disseminate whether the improvement comes from the novel views simply from additional views, we also compare with views generated along the same camera trajectory as the original views which yielded the same results as ②. We then placed additional random cameras into the scene volume to render novel views from random positions. This results in a drastic performance drop (15.4 mloU) due to numerous frames that are either inside the scene geometry or showing meaningful context leading to deteriorated open-set features. This demonstrates the benefit of our novel view synthesis approach presented in Sec. 3.3.
4.4 QUALITATIVE RESULTS FOR 3D SCENE SEGMENTATION AND OPEN-SET APPLICATIONS
In Fig. and 5, we compare qualitative semantic segmentation results of LERF, OpenScene and our OpenNeRF. The white dashed circles highlight the different predictions of each method. In contrast to OpenScene, our approach is able to correctly segment the wall-plugs as well as the blanket on the bed (Fig. 5, left). Our approach also properly detects the basket, the blinds and produces less noisy results on the door (Fig. 5, right). The more interesting aspect of open-set scene representation is that they can be queried for arbitrary concepts. In Fig. 6, we show the response of open-set queries. For each example, we provide the text query as label. We observe that our method can be used to not only query for classes, but also for concepts like object properties material types. Finally, in Fig. we show results on newly recorded "in-the-wild" sequences demonstrating the general applicability of our approach to unseen scenarios.
CONCLUSION
We presented OpenNeRF, a NeRF-based scene representation for open set 3D semantic scene un- derstanding. We demonstrate the potential of NeRFs in combination with pixel-aligned VLMs as a powerful scene representation compared to explicit mesh representations, specifically on the task of unsupervised 3D semantic segmentation. We further exploit the novel view synthesis capabilities of NeRF to compute additional views of the scene from which we can extract open-set features. This enables us to focus in greater detail on areas that remain underexplored. To that end, we pro- posed mechanism to identify regions for which novel views should be generated. In experiments OeenNeRI outperforms the mesh-based OpenScene as well as the NeRe- LERF.
Acknowledgements. This work was partially supported by an ETH Career Seed Award funded through the ETH Zurich Foundation and an ETH AI Center postdoctoral fellowship.

---

Published as a conference paper at ICLR 2024
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do as Can, Not Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:220400191 2022.
Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint DD-3ID-Semante Data for Indoor Scene Understanding. International Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. neurips, 2021.
Christopher Choy, Young Gwak, and Silvio Savarese.
Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. In International Conference Computer Viston and Pattern Recognition (CVPR), 2019.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In International Con- ference on Computer Vision and Pattern Recognition (CVPR), 2017a.
Angela Dai, Matthias Nießner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt. BundleFu- sion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration. ACM Transactions on Graphics 2017 (TOG), 2017b.
Alexandros Delitzas, Ayea Takmaz, Federico Tombari, Robert Sumner, Mare Pollefeys, and Francis Engelmann. Scenefun3d: Fine-grained functionality and affordance understanding in 3d scenes. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Knuth Donald al. The Art of Computer Programming. Sorting and searching. 1999.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling Open- Vocabulary Image Segmenta- tion with Image-Level Labels. In European Conference Computer Vision (ECCV), 2022.
Huy Ha and Shuran Song. Semantic Abstraction: Open-World 3D Scene Understanding from 2D
Vision- Languu-L Models. In Conference on Robot Learning (CoRL), 2022.
Lei Han, Tian Zheng, Lan Xu, and Lu Fang. OccuSeg: Occupancy-Aware 3D Instance Segmenta- tion. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Zeyu Hu, Xuyang Bai, Jiaxiang Shang, Runze Zhang, Jiayu Dong, Xin Wang, Guangyuan Sun, Hongbo Fu, and Chiew-Lan Tai. VMNet: VoxeI-Mesh Network for Aeeaeeec-Awaare 3D Semantic Segmentation. International Conference on Computer Vision (ICCV), 2021.
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer. Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. ConceptFusion: Open-Set Multimodal 3D Mapping. Robotics: Science and Systems (RSS), 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hien Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up Visual and Vision- LanguugL Representation Learning with Noisy Text Supervision. In International Conference on Machine Learning (ICML), 2021.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: Language Embedded Radiance Fields. In International Conference on Computer Vision (ICCV), 2023.
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for Editing via Feature Field Distillation. International Conference on Neural Information Processing Systems (NeurIPS). 2022.
10

---

Published as a conference paper at ICLR 2024
Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, and Bastian Leibe. 4D-STOP: Panoptic Segmentation of 4D Lidar using Spatio- Temporal Object Proposal Genera- tion and Aggregation. In European Conference on Computer Vision (ECCV) Workshops, 2022.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In International Conference on Computer Vision and Patlern Recognition (CVPR), 2022.
Marc Levoy. Efficient ray tracing of volume data. ACM Transactions on Graphics (TOG), 1990.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-Drin Driven Semantic Segmentation. International Conference on Learning Representations (ICL.), 2022a.
Jinke Li, Xiao He, Yang Wen, Yuan Gao, Xiaoqiang Cheng, and Dan Zhang. Panoptic-phin Towards real-time and high-precision lidar panoptic segmentation via clustering pseudo heatmap. In International Conference on Computer Vision and Paltern Recognition (CVPR), 2022b.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Pe- ter Vajda, and Diana Marculescu. Open- Vocabulary Semantic Segmentation with Mask-adapted CLIP. International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: unifying neural implicit sur- faces and radiance fields for multi-view reconstruction. In International Conference on Computer Vision (ICCV), 2021.
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. OpenScene: 3D Scene Understanding with Open Vocabularies. International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep Hierarchical Fea- ture Learning on Point Sets in Metric Space. International Conference on Neural Information Processing Systems (NeurIPS), 2017.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat- natttat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv: 109.082998888 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseClip: Language-Guin Guided Dense Prediction with Context-Aware Prompting. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
David Rozenberszki, Or Litany, and Angela Dai. Language-Groundeer Indoor 3D Semantic Seg- mentation the Wild. In European Conference on Computer Vision (ECCV), 2022.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Transformer for 3D Instance Segmentation. International Conference on Robotics and Automation (ICRA), 2023.
Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic Lifting for 3D Scene Understanding with Neural Fields. Inter- national Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica Dataset: A Digital Replica of Indoor Spaces. arXiv preprint arXiv: 057977 2019.
11

---

Published as a conference paper at ICLR 2024
Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Mare Pollefeys, Federico Tombari, and Fran- cis Engelmann. OpenMask3D: Open-Vocabulaas Vocabulary 3D Instance Segmentation. In International Conference on Neural Information Processing Systems (NeurIPS), 2023a.
Ayça Takmaz, Jonas Schult, Irem Kaftan, Mertcan Akçay, Bastian Leibe, Robert Sumner, Francis Engelmann, and Siyu Tang. 3D Segmentation of Humans in Point Clouds with Synthetic Data. In International Conference on Computer Vision (ICCV), 2023b.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kell Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works learn high frequency functions low dimensional domains. International Conference on Neural Information Processing Systems (NeurIPS), 2020.
Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representations. In International Conference on 3D Vision (3DV), 2022.
Can Wang. Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. CLIP-NeRF: Text-and- Image Driven Manipulation of Neural Radiance Fields. In International Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for nueti-view reconstruction. In Interna- tional Conference on Neural Information Processing Systems (NeurIPS), 2021.
Samuel S Wilks. Certain Generalizations in the Analysis of Variance. Biometrika, 1932.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In International Conference on Neural Information Processing Systems (NeurIPS), 2021.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Ex-
ploring monocular geometric cues for neural implicit surface reconstruction. In International Conference on Neural Information Processing Systems (NeurIPS), 2022.
Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries. In International Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place Scene La- beling and Understanding with Implicit Scene Representation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Os- wald, and Marc Pollefeys. Nice-SLAM: Neural Implicit Scalable Encoding for SLAM. In Inter- national Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
René Zurbrügg, Yifan Liu, Francis Engelmann, Suryansh Kumar, Marco Hutter, Vaishakh Patil, and Fisher Yu. ICGNet: A Unified Approach for Instance-Centric Grasping. In International Conference on Robotics and Automation (ICRA), 2024.
12


Reviews:
[{"review_id": "ua8o7e9n4j", "reviewer": "Reviewer_Nxv7", "summary": "This work explores a similar paradigm to LeRF (Language-embedded radiance fields). Rather than distilling CLIP features to NeRF, rather it uses a different backbone i.e. OpenSeg to distill OpenSeg features to the 3D using NeRFs. The idea of distilling similar open-set features to 3D has been shown by OpenScene which required 3D meshes as input 3D representation, however, the paper studies it in the setting of Neural Radiance Fields which requires posed 2D images as input. Qualitative open vocabulary segmentation comparisons are shown on the Replica dataset coupled with quantitative comparisons with baselines like LeRF and OpenScene", "questions": "Please see all the questions in the weakness section. Overall I would have liked to see a thorough comparison with LeRF and what technical improvement the work bring (in addition to changing the pre-trained backbone), a comparison with Semantic-NeRF's uncertainty formulation, more in the wild examples and examples where the quality of NeRF is not that good and do the semantic features break or does uncertainty help in those areas", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The approach discussed very relevant problems i.e. distilling open-set 2D features to the 3D domain and moving away from traditional pipelines that work with a known number of categories i.e. approaches like Semantic-NeRF and Panoptic NeRF etc. The main benefits/strengths of the approach are as follows:\n\n1. Qualitative comparison with LeRF and Open-Scene clearly shows better segmentation results despite the fact that LeRF was not designed for segmentation (which should be clearly pointed out)\n\n2. Clearly better quantitative results on all, head, common, and tail sets\n\n3. A good measure of uncertainty to improve the open-set feature field distillation into 3D.", "weaknesses": "1.  Though the uncertainty measure is sound, I wonder if this improves LeRF and OpenScene's performance as well. A similar measure was introduced in Semantic-NeRF and the authors didn't highlight the difference between their formulation and Semantic NeRF's formulation. \n\n2. This looks like an incremental work that extends LeRF by using a different encoder backbone which is very straightforward to implement. Can the authors justify it with really good segmentation performance on long tail in-the-wild queries etc? I didn't see that comparison\n\n3. Not many in-the-wild examples could be seen in the paper. Replica dataset is easier since we have perfect viewpoint annotations. I wonder if the performance stays steady or breaks for more in the wild examples where there are imperfect viewpoint annotations. \n\n4. Does uncertainty measure really help? What if the initial NeRFs are not that good? Features tend to break for those viewpoints/areas. Do the authors have improved results in those cases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "73Ttd4eMB1", "reviewer": "Reviewer_Y2vi", "summary": "This paper introduces OpenNeRF, an innovative approach for superior open-set 3D semantic scene understanding. OpenNeRF demonstrates its suitability for detecting small long-tail objects, surpassing mesh-based representations in performance. Additionally, OpenNeRF effectively represents interesting scene parts, leading to improved segmentation performance.", "questions": "1. Could you provide more details on the evaluation protocol used to compare explicit-based (mesh/point cloud) and implicit-based (NeRF) methods for open-set 3D semantic segmentation? What were the key findings of this evaluation?\n2. Have you considered the scalability and computational efficiency of OpenNeRF? How does it perform in terms of runtime and memory requirements compared to other methods?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed OpenNeRF approach significantly improves 3D semantic segmentation results compared to baseline methods such as LERF and OpenScene. It can also detect challenging long-tail classes ignored by other methods.\n2. Confidence estimation and analysis of novel views bring new insights for improving the Open-set 3D scene understanding.\n3. OpenNeRF can be queried for arbitrary concepts, including object properties (eg. reflective, soft) and material types (eg. wood, cotton), showcasing its versatility. It shows impressive results in the wild scene using a phone scanner.", "weaknesses": "1. The performance drop when rendering novel views from random positions suggests that the approach is sensitive to the quality and meaningful context of the additional views.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WLVw8ePS1", "reviewer": "Reviewer_UKPx", "summary": "The paper presents a NeRF-based scene representation for open-set 3D semantic scene understanding. The method extends previous work by leveraging pixel-aligned features and a mechanism to identify regions that require generating novel views. These additional views of the scene allow the proposed approach to extract more open-set features and improve the overall understanding of the 3D semantic scene. The results have been tested on the Replica dataset and have outperformed a mesh-based baseline and a NeRF-based baseline.", "questions": "Please see the weaknesses part for my concern and confusion.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper is well-written and justified. All the technical details are well-elaborated.\n* The concept of using novel view synthesis capabilities to extract additional visual-language features for better scene understanding is interesting and straightforward.\n* The qualitative results appear satisfactory and the quantitative results outperform OpenScene and LERF.", "weaknesses": "My main concern is regarding the evaluation:\n* While the Replica dataset may be challenging for long-tailed settings, it would be great to evaluate on larger-scaled datasets and outdoor benchmarks such as Matterport3D, ScanNet, and nuScenes.\n* One of the main contributions of this work is the use of novel view synthesis to extract additional visual-language features. To ensure the quality of the learned features, it seems necessary to evaluate the quality of the rendered views and their effect on scene understanding.\n* To better evaluate the proposed method, it would be better to compare it with 2D methods such as LSeg, OV-Seg, and ODISE, as well as other 3D methods including Feature Field Distillation, Semantic-NeRF, and Panoptic Lifting (disregard whether these methods are open or closed-set). \n\n\nSome questions regarding the confidence estimation and novel camera view selection:\n* The uncertainty map in Figure 3 indicates that the door and cabinet are the most uncertain points, while the generated novel pose in Figure 2 seems to focus primarily on the table. Given high uncertainty points, how are the camera views sampled?\n* Although there is a positive correlation between per-point uncertainty and per-point error, the heat maps appear to emphasize all areas other than the wall and floor. It is also intriguing that the peaks of uncertainties don't correspond to object boundaries, but rather the entire object. It would be better to demonstrate more samples to showcase these findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v7tsjMj3HR", "reviewer": "Reviewer_AABJ", "summary": "This paper proposes to address the task of 3D open set detection by harassing implicit reconstruction methods, and specifically NeRF.\nIf I am not mistaken, the method boils down to adding supervision/prediction-head of per-pixel CLIP features in addition to the RGB/color head, while keeping the per-voxel density $\\sigma$ prediction used in NeRF. \nThe authors use depth supervision, when available, as done in previous works.\n\nThe devil lies, as always, in the details of the training regime, and the authors stress some details that are important to make the network converge to the presented results:\nThe authors use OpenSeg feature maps which are more localized and therefore do not supervise them in a multi-scale fashion.\nAdditionally, one should refrain from using such features near the image edge as they are less stable.\nFinally, one of the main points in the training process, the authors claim that training on novel views (generated by a trained NeRF) makes as big difference. Specifically, novel views are selected to minimize the uncertainty of the OpenSeg features, measured as generalized variance on the in 3D. This help mitigate areas where open-set features \"disagree\" from multiple viewpoints.\n\nThe experimental section evaluates the method compared to the previous 3D open-set approaches, which are all explicit.\nThe authors divide the labels into 3 groups by frequency, which better emphasized the strength of the propose method on rare ones.", "questions": "A question on the novel-view based training: \nCan you please comment on the seemingly bootstrapped nature of this approach?\nIn other words - a NeRF model has to be trained in the first place to generate novel views, and these novel views are then trained to predict open-set features?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper was rather easy to follow, for an experienced NeRF reader (which is reasonable these days :-) ).\n* The base of the method is rather straight forward to understand (adding CLIP predictions)\n* The authors indicate uncertainty estimation of the open-set head is  a strong signal to help the network converge, with help of NeRF novel views.\n* The results show the advantage of implicit methods over explicit ones\n* The ablation study helps to understand which part of the training regime made the difference.", "weaknesses": "Many of the details of the paper were presented in previous works (e.g. per-pixel feature supervision was presented in NeRF-SOS fo DINO). and a big part of the novelty here is in the details of applying them to 3D segmentation.\nThis alone does not prevent acceptance, IMHO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)