Decomposing NeRF for Editing via Feature Field Distillation
Authors: Sosuke Kobayashi
Eiichi Matsumoto
Vincent Sitzmann
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural radiance field
feature field
editing
semantic segmentation
zero-shot segmentation
distillation

Abstract: Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations.
However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional.
In particular, it has been difficult to selectively edit specific regions or objects.
In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes.
We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field.
Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training, and enables us to semantically select and edit regions in the radiance field.
Our experiments validate that the distilled feature fields can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

Decomposing NeRF for Editing via Feature Field Distillation
Sosuke Kobayashi Preferred Networks, Inc. sosk@preferred.jp
Eiichi Matsumoto Preferred Networks, Inc. matsuuoto@preeerre.. jp
Vincent Sitzmann
Massachusetts Institute of Technology si tamannnmii edu
pfnet-research thub
Abstract
Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a the underlying connectionist representations such as MLPs
NeRF is challenging, or voxel grids are not object-centric or compositional. In particular, has been difficult selectively edit specific regions objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 31) scenes. We propose distill the knowledge of off-the-shelf, supervised and self-supervised 21) image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given user-specified query of various modalities such as text, an image patch, point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.
Introduction
Emerging neural implicit representations or neural fields have been shown be a promising approach for representing variety of signals [82, 53, 65, 106, 56). In particular, they play an important role 3D scene reconstruction and novel view synthesis from a limited number of context images. Neural radiance fields (NcRF) [56] enabled the recovery of continuous volume density and radiance field from a limited number observations, producing high-quality images from arbitrary views via volume rendering with promising applications in computer graphics. However, editing scene reconstructed by NeRF is non-obvious because the scene is not object-centric and is implicitly encoded in the weights of connectionist representation such an MLP [56] voxelgrid [23]. Although we can transform the scene in input or output space or via optimization based editing [37, 97], this semantic, local edits, such moving single object.
does not enable selective object-centric Prior work has addressed this challenge via coordinate-level. semantic decompositions which allow to selectively move, deform, paint, optimize parts of NeRF, but relies costly annotation of instance segmentations and training of instance-specifer networks [104]. While this can be alleviated with pre-trained segmentation models [25, 41], such models require pre-defined closed label sets and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

domains (e.g.. traffic scenes), limiting decomposition and editing. Local editing of NeRFs ideally requires an efficient, open-set method for coordinate-level decomposition.
In this work, we present distilled feature fields (DFFs), a novel approach to query-based scene decomposition for local, interactive editing of NeRFs. We focus on 3D neural feature fields, which map every 3D coordinate to a semantic feature descriptor of that coordinate. Conditioned on a user query such as a text or image patch, this 3D feature field can compute decomposition of a scene without re-training. We train a scene-specific DFF via teacher-student distillation [34], using supervision from feature encoders pre- pre-trained on the image domain. Unlike the domain of 3D scenes, the image domain boasts massive high-quality datasets and abundant prior work on self-supervised and supervised training of effective feature extraction models. Notably, recently proposed manssfomerrbbasee models [96, 22] have demonstrated impressive capabilities across various vision- and text- tasks (e.g., CLIP [69], L.Seg [44], DINO [12]) Such feature spaces capture the semantic properties of regions and make it possible to correspond and segment them well by text, image queries, or clustering. We employ these models as teacher networks and distill them into 31) feature fields via volume rendering. The trained feature field enables to semantically select and edit specific regions in 3D NeRF scenes and render multi-view consistent images from the locally edited scenes.
In extensive experiments, we investigate the applications of neural feature fields with two different pre- pre-trained teacher networks, (1) LSeg [44], a CLIP-inspired kunuuag--rrive semantic segmentation network, and (2) DINO [12, 3], a self-supervised network aware of various object boundaries and correspondences. LSeg and DINO features allow us to select 3D regions by simple text query or an image patch, respectively. We first quantitatively demonstrate that LSee- based DFFs with label queries can have high 3D segmentation performance compared with an existing point-cloud based 3D segmentation baseline trained on ScanNet [20], a supervised point-cloud dataset. We then demonstrate a variety of 3D appearance and geometry edits across real-world NeRF scenes with no annotations of segmentation; and show that we may edit regions with a single query of text, image, pixel, or cluster choice.
2 Related Work
Neural Implicit Representations. Neural implicit representations or neural fields have recently advanced neural processing for 3D data and multi-view 2D images [82, 53, 65, 106, 56]. For a review of this emerging space we point the reader to the reports by Kato et al. [39]. Tewari et al. [90], and Xie et al. [102]. In particular, a neural radiance field (NeRF) can be fitted to a set of posed 2D images and maps a 3D point coordinate and view direction to RGB color and density. When observations are limited, NeRF often overfits and fails to synthesize novel views with correct geometry and appearance. Pre-trained vision models have been used for regularizing NeRF via flows [62], multi-view consistency [35], perceptual loss [1 [1]] or depth estimation [100, 77J. Some pre-trained models operate not only in the visual world but also in other modalities such language. The recently proposed CLIP model [69] has demonstrated impressive performance in image-and-text aligmment, with strong generalization to various textual and visual concepts. Wang et al. [97], Jain et al. [36], and Poole et al. [68] use CLIP or Imagen [79] to edit or generate single-object NeRF with a text prompt query by optimizing the NeRF parameters to generate images matched with the text. While such methods are promising, they do not enable accurate selective editing of specific scene regions. For example, the prompt "yellow flowers" may affect unintended scene regions, such as the leaves of a plant. Our proposed decomposition method leverages pre-trained foundation models to enable selective editing of real-world NeRF scenes. Neural descriptor fields [80] use intermediate features that emerge in a 3D occupancy field network [53] for efficiently teaching robots object grasping. Instead of a pre-trained object-centric 3D model, we use 2D vision models as teacher networks via distillation, exploiting recent progress in pre-trained foundation models [7].
Geometric Decomposition of Neural Scene Representations Kohli et al. [40] and Zhi et al. [112] show that neural implicit representations can be combined with the supervision of semantic labels. Yang et al. [104] demonstrate that given view-consistent ground-truth instance segmentation masks during training. NeRF can be trained to represent each object as different volumes, although such an annotation is expensive in practice. Concurrently, Benaim al. [6] also experiment with the different parametarization. Conditional [49, 37, 21, 63] and generative models [60, 61, 31] enable degree

---

of category-speciff decomposition (e.g,, human bodyparts) and editing on constrained domains with large datasets. Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters. However, the decomposition limited due to the inflexibly structured boundaries or strong assumptions about scenes; self- object-centric learning is difficult task. Other studies also explored reconstruction with more structured hybrid representations via pipelines specialized to domain (e.g.. traffic scene) [64, 25, 41] or situation (e.g., each object data is independently accessible) [28, 27, 105]. Note that this line work defines and constrains domains the types of segmentation during before training and thus limits the degrees of freedom for editable scenes and objects. In contrast, our method can decompose scene- specifcc NeRFs into arbitrary semantic units via text and image queries, enabling versatile scene edits without re- training. A concurrent paper by Tschernezki et al. [93] also explores the same training framework and, in particular, investigates how fused features are improved from 21) teacher networks. It also complementarily shows the results with other teacher models (MoCo-v3 [17] and DeiT (91]),
dimension reduction via PCA, and NeuralDiff [92]-based neural fields. Other concurrent studies explore decomposition through training scene-specific segmentation field [113] or 3DCNN [76] supervised by click scribble annotations. Lastly, in different but related task, video editing, Kasten et al. [38] use
decomposition and atlas representation for time-consistent, local editing; Loeschcke et al. [51] and Bar-Tal& et [5] further use CLIP for editing.
Zero-shot Semantic Segmentation. Zero-shot semantic segmentation is challenging task [24, 2, 10] where a model has to predict semantic labels of pixels in images without a a-priori information of the categories. A typical solution is to vision-and-languue trained to encode images (pixels) and text labels into the same semantic space and perform zero-shot prediction based on the similarity or alignments of the two inputs. Recent development of image encoder architectures [96, 22, 71] and large- scale training [69. 12] have improved the ability and generalization of vision models, including zero-shot models [44, 52, 99, 103, 114, 72]. On the other hand, ongoing studies on zero-shot perception in 3I) still suffer from the lack of effective, efficient, and high-resolution architectures and large-scale annotated datasets [54, 33, 29, 101, 78, 30J. Our method is a new approach to perform zero-shot semantic segmentation on scene-specific 3D fields by exploiting progress in the image domain without semantic 3D supervision. We note that the goal of this paper is not to achieve state-of-the-an performance on 3D semantic segmentation tasks. Instead, our goal is the decomposition of neural scene representations for editing, which requires smooth segmentation results on continuous 3D space rather than segmentation of discrete point clouds or voxelgrids.
cross-modal encoders. They are
Preliminaries
3.1 Neural Radiance Fields (NeRF)
NeRF [56] uses MIPs to output density a and color given point coordinate (x, v. 2) in a 3D scene. This simple scene representation can rendered and optimized via volume rendering. Given pixel's camera ray r(t)- id, depth & with bounds [fuear, 4tar. camera position and its view direction d, NeRF calculates the color ray using quadrature of K sampled points { with depths (t K as
K C(r) = Σ f(tx) (o(x))6N) c(xx) d), f(tx) exp k=1
k-1 Σ Ø=1
ox (xx)))x)
(I)
where (x: 1- exp(-x)) and dR. le+1 is the distance between adjacent point samples. NeRFs are optimized solely on a dataset of images and their camera poses by minimizing re- rendering loss.
3.2 Pre-trained Models and Zero-shot Segmentation of Images
Most semantic segmentation models pre-define a closed set of labels, and cannot flexibly change the segmentation categories or boundaries without supervised training. In contrast, zero-shot semantic segmentation predicts target regions given open-set queries. Li al. [44] proposes LSeg, model to
3

---

Training
(x)
volume rendering
minimize differenee F(r)
Decomposition
query
fingl (I.rr
d x
l (0)
(x) f(x) c(x, d)
(((xx)
fimg
(())
C(r)
c(x,
PE(x)
PE(d)
Figure 1: Left: A Distilled Feature Field (DFF) maps a coordinate and viewing direction d to density a, color c, and feature f. It is trained by minimizing the difference between rendered features and features as predicted by pre-trained image feature encoder, as well as the rendered color and ground-trutt pixel color. Right: At test time, we may decompose and edit 3D space via selecting and manipulating different 3D regions with variety of queries.
perform zero-shot semantic segmentation by aligning pixel-level features and text query feature. LSeg employs an image feature encoder with the DPT architecture [71] and a CLIP-based text label feature encoder [69], trained via large-scale language-image contrastive learning. The probability of a text label given pixel r in an image p([]T,)), is then calculated via dot product of pixel-Ievel image feature fing r), and queried text feature (1) followed by softmax:
exp(fing (1, (()))
p(I "r (L)
(2)
where C is set of possible labels. If negative labels thresholded cosine similarity to directly compute the probability of a label. During training, LSeg optimizes only the image encoder (I, by minimizing cross-entropy on supervised semantic segmentation datasets. The text encoder (4) is obtained from pre-traines CLIP model [69]. Recently, pre-trained CL.IP has been leveraged as the backbone for variety of tasks and has been extended with additional modules sharing the same latent space. For example, Reimers and Gurevych [74, 75] trains a multi--inguul (more than 50+ languages) text encoder, which enables CLIP and CLIP "LIP-inspired variants to use non-English queries like Japanese. We similarly use the latent space of a pre-trained CLIP for LSeg via distillation, enabling the decomposition of NeRFs with both English and non-English queries. Segmentation can further be performed with other modalities such as image, patch or pixel query features f using a similar dot-product similarity formulation as in Eq. Notably, DINO [12], self-supervised vision model, solves video instance segmentation and tracking by calculating similarity among features in adjacent frames. Amir et al. [3] also demonstrate that DINO features work well on co-segmentation and point correspondence by similarity and clustering. In our experiments, we use these two publicly available models, LSeg and DINO, to obtain features of images and texts for 3D decomposition.
available, we may use other scores like
4 Distilled Feature Fields
4.1 Distilling Foundation Modules into 3D Feature Fields via Volume Rendering
NeRF learns neural field to compute the density and view-dependent color, oxx) and c(x, d). We may extend NeRF by adding decoders for other quantities of interest. For example, SemanticN- eRF [112] adds a branch outputting a probability distribution of closed-set semantic labels, trained with supervision via images with ground-truth semantic labels. This enables the prediction of pairs of RGB and semantic segmentation masks from novel views, useful for data augmentation. However, because ground-truth annotation is costly, the method is inefficient as a means of scene editing [104]. For specific domains like traffic scenes [25, 41], we may instead train a closed-set segmentation model and use its prediction for training object-awane neural fields. However, this approach is possible only the types of objects are limited and the comain-specifie supervised dataset is available; limiting the application of scene editing in terms of domain and flexibility of decomposition.

---

We build on top of these ideas and perform 3D zero-shot segmentation of NeRFs using open-set text labels or other feature queries. Instead of a branch performing closed-set classification, we propose to add a feature branch outputting feature vector itself. This branch models a 3D feature field describing the semantics of each spatial point. We supervise the feature field by pretrained pixel-level image encoder f fing as a teacher network. Given 3D coordinate x, the feature field outputs a feature vector f(x) in addition to density o(x) and color c(x, d), as shown Fig. Volume rendering of the feature field is similarly performed via
F(r) Σ ((((x)))) f(xx).
We can optimize f by minimizing the difference between rendered features F(r) and the teacher's features fing (T. Effectively, we are distilling [34] the 2D teacher network into our 3D student network via differentiable rendering, and thus dub this model distilled feattue field (DFF). We add feature objective f, penalizing the difference between rendered features F(r) and the teacher's outputs fing (4, to the photometric loss of the original NeRF. We use two networks for volume rendering with poarse-and-fing hierarchical sampling. We thus minimize the sum of photometric loss Lp and feature loss Ly. in total, L:
L + Lp = Σ (((r)- (00()-col)) FER
L, Σ F(r) FER
fung (
(4)
where R are sampled rays, C(r) the ground truth pixel color of ray r, À is the weight of the feature loss and is set to 0.04 to balance the losses [112], We apply stop-gradient to density in rendering of features F(r) in Equation 3 as the teacher's features fing r) are not fully multi-view consistent, which could harm the quality of reconstructed geometry.
4.2 Query-based Decomposition and Editing
A trained DFF model can perform 3D zero-shot segmentation by its feature field f and query encoder f. Probability of a label I of point the 3D space, p([|x), is calculated by dot product of the 3D feature f fxx and text label feature (1) followed by softmax:
exp(((x))
p([x) = -rrc (())
(5)
This query- based segmentation field is at the core of the proposed method. It can be calculated at any 3D point without limiting resolution, naturally used in tandem with radiance field and volume rendering. Note that the segmentation depends on only the 3D coordinate and the query'. As the original NeRF, is thus multi-view consistent. In addition and important for interactive editing, we can change the segmentation via queries without re-training, which cannot be realized by closed-set methods using semantic [112] or instance segmentation annotation [104]. We may now use this query cecrycooniiional segmentation to identify specific 3D region for editing. Various edits can be generalized to the merging of two NeRF scenes 0 (x), c (x, d) and (x()), (x d), where we use the segmentation field p for blending. In the experiments section, we simply modify Eq. I as a blend of two scenes based on the ratio of a:
K
C(r) Σ f(tx) (a(a1 ei (x d)pk  a(oz(xx)6)) (x) cx d)(1 px))
(6)
k=1
(x+))))
k-1 II a (or (7)
where Pk alo (xx) +
f(ta)
==1
For example, if we want apply geometric transformation g region of a query / in a NeRF
scene (a, c), we can render the transformed scene via Eqs. 6 and 7 by setting a(a (xx) )6p) = (I p(((x)))a( (oxx())))(()))))) alaz (x )o+) = p('lg " ( ((xx))8x), ea (x2, d) (1
d), and C2 d) = p - (xx (xx), (d)). More details editing for
'It is an interesting extension to introduce user's viewpoint to the function for recognizing view deeedeeen queries like referring expressions (e.g.. "the chair left to the table") [16, 1, 47, 4]. We leave this to future work.
5

---

coarse MLP
Table I: Performance of 3D semantic segmentation on Replica dataset. DFF outperforms supervised point-cloud segmentation model MinkowskiNet42.
Ground truth
Edstars
mIoU
accuracy
Ladee
fine MLP
Supervised 3DCNN
0.475
0.758
Tlc
DFF (Coarse)
0,589
0.855
Figure 2: Comparison of predictions by coarse and fine MLPs.
DFF (Fine)
0.583
0.855
colorization, translation, and deletion are shown in Appendix B. We can combine this with more complex edits, including optimization-
methods like CLIPNeRF [97]. While CLIPNeRF itself cannot selectively edit specific regions in multi-object scenes, our decomposition method enables it to update only desired objects without breaking unintended areas.
5 Experiments
We first conduct a quantitative evaluation of the decomposition achieved by DFF. We demonstrate that DFF enables 3D semantic segmentation in a benchmark dataset using scanned point clouds with numan-annotated semantic segmentation labels. We then investigate the capabilities of DFF for editing and subsequent novel-view synthesis on real-world datasets. We use two teacher networks, LSeg [44] and DINO [12), which are pre-trained and publicly available. Each training image is encoded by the image encoders of the networks and used as target feature maps, fing (Z, r), defined in Equation 4. Because the feature maps are of reduced sizes due to the limitation of the networks, first resize them to the original image size. The implementation and settings of NeRF, unless otherwise stated, follow Zhi et al. [112]. During the training of 200K iterations, the loss in Equation 4 is minimized by Adam with a linearly decaying learning rate (5c-4 8c-5). During training, Gaussian noise for density also applied. The number of coarse and fine samplings is 64 and 128, respectively. The MLP of the neural radiance field consists of eight ReLU layers with 256 dimensions, followed by linear layer for density, three layers for color, and three layers for feature, as shown in Fig. Positional encoding of length 10 is used for the input coordinate and its skip connection, and that of length 4 is for viewing direction. If an independent MLP is prepared for the feature field, it consists of four layers (with skip connection at the third layer if the positional encoding is used). The size of training image is 320 x 240 for the Replica dataset and 1008 x 756 for the other datasets. The batchsize of training rays is 1024 for Replica and 2048 for the others. During finetuning of feature fields or radiance fields, Gaussian noise is removed, and the learning rate is set le-4. See appendix and C for further training details.
5.1 3D Semantic Segmentation
We construct a 3D semantic segmentation benchmark from four scenes in the Replica dataset [86] with data split and posed images provided by [112]. See appendix D for further details of the dataset. We train DFF reconstruct each scene with radiance and feature fields from training images and evaluate the quality novel view synthesis and 3D segmentation of the annotated point clouds. We use LSeg as teacher network. The LSSeg text encoder encodes each label, and the probability of each point calculated by Equation 2 Note that the training uses only the photometric and feature losses (Equation 4) and does not access any supervision via semantic labels.
Semantic Segmentation Results. First, we show evaluation metrics of 3D semantic segmentation, mean antersection-over-mmee (mloU) and accuracy in Table For comparison, we also experiment with a sparse 3D convolution-based segmentation model, MinkowskiNer42 [18] taking a colored
While LSeg- -FF can perform zero-shot inference using text labels are not seen during training, we do not focus on thoroughly evaluating the zero-shot ability. The evaluation has been conducted the original paper on the teacher network, and DFF's ability is expected to follow it due to distillation. Please refer to Li et al. [44] for the detail of the zhoo ability of LSeg.

---

Table 2:
Performance of novel view synthesis on Replica dataset. PSNR, SSIM, and LPIPS are metrics of image synthesis. 5 <1.25 and absrel are metrics of geometry (depth estimation).
PSNR
SSIM
LPIPS|
5 5<1.25
absrell
basic NeRF
32.87
0.934
0.148
0.993
0.018
DFF
32.85
0,932
0.150
0.993
0.017
DFF (overweighting X)
32.68
0.927
0.162
0.993
0.018
point cloud as input. It has a standard state-of-the-ard architecture for point cloud segmentation and is trained on the ScanNet dataset [20], the largest annotated training dataset of 3D semantic segmentation'. Results demonstrate that DFF, taught by LSeg, achieves promising performance, even better than the supervised model. This indicates that DFF succeeds at distilling 3D semantic segmentation from the 2D teacher network.
Impact of Sampling on Semantic Segmentation. NeRF employs two MLPs for hierarchical sampling, where the coarse MLP performs volume rendering with fewer points (64) using stratified sampling, and the fine MLP works with importance sampling (192 in total). So, we have two sampling options to train a feature field. Although fine sampling critical for training accurate radiance fields, segmentation is of significantly lower spatial frequency than texture. We thus analyze the impact of coarse and fine training in Fig. 2. As expected, the coarse model produces smooth segmentations. while the fine version introduces high-frequency artifacts. This smoothness property is important for natural editable novel view synthesis and is discussed again later.
Compatibility with View Synthesis. We also check and compare the quality of novel view synthesis with NeRF, which does not learn feature fields. Because the feature branch partially shares the layers with the radiance field (as shown in Fig. 1), learning feature fields could possibly harm the radiance field. Despite this concern, as shown in Tab. 2, the performance of view synthesis is not degraded. Thus, we can train and use the branch-based DFF with small computational and parameter overhead compared to the original NeRF. If we excessively increased the weight of the feature loss, À x 10, hurt view synthesis while not improving segmentation performance further. We further confirm that training independent, light-weight feature-field MLP, instead of a branch of the radiance- field MLP, achieves semantic segmentation results competitive with the branch-based approach (see appendix Tab. for the result of all variants)*. This option is useful especially when we want to introduce DFF decomposition into arbitrary 3D scene representations, including off-the-shher NeRF models, dynamic NeRFs [26, 66, 45], or meshes, without re- trainiiiig of the radiance field.
5.2 Editable Novel View Synthesis
In the previous section, we quantitatively validated the ability of DFF to perform semantic decom- position. We now discuss the capability for editable view synthesis on real-world scenes, including the LLFF dataset [55] and our own dataset. Our method can be used even for LLFF scenes based on normalized device coordinates. Please see the supplemental web page for further results, including videos. In addition to LSeg using a text query, we also experiment with self-supervised DINO [12] as another teacher network to enable query- use thresholded cosine similarity to directly compute the probability of query instead of softmax with negative queries in Eq. 5 and set p = 1 if the similarity excceds the threshold 5 and p = 0 otherwise for hard decomposition. We first train NeRFs without feature branch for each scene for 200K iterations (Lp) and then finetune them with a feature branch via distillation for 5K iterations (Ip + AL since we found that the feature loss converged significantly faster than the photometric loss and short training was thus sufficient. We use coarse sampling for training feature branches and use it for edited rendering with fine sampling. See appendix Á for the details.
decomposition using image patch queries. Here, we
'Frr a fair comparison, the label set follows the ScanNet dataset. We also manually tune the range and scale of input point clouds to maximize the performance of MinkowskiNet42 on the Replica dataset.
*Note that, even if we are interested in the feature-fiel MLP, it is important to simultancously prepare the radiance-fiel MLP for reconstructing geometry well and training the feature field via geometrically plausible volume rendering.
'The results of the LLFF room scene used label set, |whiteboard, ceiling, light, television, wall, bin, table, cabinet, cable, chair, box, floor), for avoiding tuning of thresholds.
7

---

i
patch
"I "flower"
"light"
"chair"
"television"
"florr"
Figure 3: Appearance edits of specific objects via different query modalities: an image patch or text.
Observed view
Novel view
Feature
① delete
W
extruct
delete
nma 3 delete
Inpainting
Queriest
potele
① patct:
"Rowee"
Figure 4: Extraction and deletion of specific objects via different query modalities, an image patch or text. The edited views are 3D consistent, unlike an image inpainting baseline [87]
Appearance Editing, Deletion, Extraction. We show qualitative evaluations of novel view syn- thesis in Fig. 3 and Fig. 4. Specific 3D regions in these scenes are identified and locally edited via decomposition depending on various query modalities. In these experiments, we use text query for LSeg-DFF DFF as in Section 5.1 and use an image patch query for DINO-DFF Because DINO features capture the similarity and correspondences of regions well thanks to sup--vuee learning [12, 3], image patch queries help select all semantically similar areas at once. The patch feature is then calculated by averaging the features of pixels the patch.
In Fig. 3, we demonstrate that the DFF enables convincing selective appearance edits. Because our focus region selection via decomposition, we use simple color transformation for clarity here (e.g., flip RGB to BGR, blend colors). One might think that the MLP of radiance field by the original NeRF also has hidden layers, and their features could possibly be used for decomposition. We confirm that the naive usage of NeRF features is not robust to decomposition, as shown in Fig. 5, especially in complex multi-object scene. We use the 8th hidden layer of the fine radiance field network (i.e., the layer just before branching in Fig. 1) NeRF features cannot clearly decompose even objects with simple shapes and colors. The region selections are leaked other parts with similar colors, geometry, or positions while they do not entirely cover the targets. For example, floor selection is leaked to walls, table, bins, or ceilings. Chair selection is leaked to irrelevant black parts like television, cables, lighting equipment, or shadows. This indicates that the feature space of the original NeRF does not learn semantic similarity well and is entangled with unpredictable and more low-level factors like color spatial adjacency.
In Fig. 4, demonstrate that the DFF also works well on deletion or extraction of objects, using two patch queries (query-Q) for leaves and ground, query-Q) for flowers) and text query- "flower". For comparison with a baseline editing method, we show the results by a -tate-of-thh-arr image inpainting model, LaMa [87]. Because the model requires masks for impainting regions, we manually annotate the views for evaluation. As shown in the figure, the image inpainting model cannot generate clear and realistic images, and the different views are inconsistent. On the other hand, DFF produces multi-view consistent plausible results, especially succeeding at extracting foreground objects. Although the performance deleting foreground objects is high, a remaining shortcoming is the existence of floating artifacts and blurred volumes the far distance behind the deleted object.
Priors for Smooth Decomposition We can organize the challenges of editable NeRFs into several categories: surface decomposition, volume decomposition, lighting decomposition, and estimation of less or never observed parts. If we edit appearances only, it practically requires decomposing regions only near the surface of objects, i.e., surface decomposition. because the color of a ray is determined mostly in condensed interval around the surface. On the other hand, geometric transformations often
Other layers the coarse MLP of the NeRF also indicated similar behaviors but little qualitatively.

---

Using NeRF' feature
Our method
low threshold
high threshold
"floor"
RGB
Features
Delete fune
clusters
coarse
panch
No Branch-basat
chair
parsk
Query putchest
Features
Figure 6: Comparison of predictions by branch-based feature field MLP and indepen- dent MLP with positional encoding, each of which is trained with coarse and fine sam- pling.
Figure 5: Appearance edits of specific objects, compared with decomposition using features of NeRF hidden layer. For reference, we also show PCA-based visualizations of the features.
require a higher level of decomposition. As shown in the deletion examples, geometric transformation may move or remove some surfaces and expose the space behind them. This forces models render unknown regions less or never observed due to occlusions, including even the inside of objects. Thus, it is desirable decompose volumes smoothly while synthesizing their inside and back Although these include the same challenges as novel view synthesis tackles, editability further highlights their importance.
Apart from lighting decomposition discussed in prior work [8, 9, 111], further investigate the new chal- lenge of smooth volume decomposition by experi- menting with different DFF setups. As discussed in Section 1. DFF has two sampling options to train feature fields. The coarse training may introduce smoothness regularization and help cohesive decom- position and smoother in- in-painting of unobserved re- gions. Another reasonable smoothness regularizer is to eliminate the high-frequency positional encoding (PE). We thus train an independent MLP network for a feature field without PE. We compare four combi- nations of renderings in Fig. 6. To better understand their behavior, we use the DINO-DFF, show k-means clusters the rendered feature map, and delete the head of the Triceratops by query choosing its corresponding clusters. As expected, coarsely trained models and no-PE models succeed in smoother volume decomposition, and this combination can minimize higu-freyuency floating artifacts. A side effect is the lack of high-frequency representa- tion power, which sometimes deletes disparate background regions and misses to represent features of complex structures (c.g., the cluster visualization of the thin frames of the window). Towards the best of both worlds, developing proper priors or inductive biases is an important direction for future work [70]. Otherwise, surface-awan representations like IDR [106, 98] could avoid problems with floating artifacts. Note that not all geometric edits suffer from these problems. For example, often less problematic to move objects closer to the camera, enlarge them, warp them other scenes, as shown in Fig.
"gpl:" "aaaaa"" "cant
"gel""
Figure 7: Editing with warping, deformation, shift, and rotation.
Localizing Optimization-basee Editing. Finally, we show a combination with an optimization- based editing method. CLIPNeRF [97] optimizes the parameters of a radiance field so that its rendered images match with text prompt via CLIP. While it is mainly designed for single-object scene of specific categories, is possible to apply other real- worldw NeRFs. However, because it cannot control the scope of editing, prompt like "white flower" may change the color of unintentional targets like leaves. Our based decomposition can upgrade such an optimization-base method
"ooe that this problem also arises when Yang et al. [104] used ground-truth instance segmentation masks and trained multiple networks, although the authors did not investigate this issue.

---

Raw NeRF rendering
CLIPNERF "white flower"
Composition CLIPNeRF + our DFF-based method
"white flower"
"yellow flower"
"rainbow flower"
"suntlower"
"petunia"
Figure 8: Comparison of appearance editing by CLIPNeRF and our extension.
to render scene via the composition of CLIP-optimized NeRF scene and the original NeRF scene. We show the results Fig. 8*. Although the naive CLIPNeRE edits unintentional parts, our method helps it to locally edit intentional parts only. In addition switching rendering, we can also use the decomposition for controlling training signals during backpropagation. The additional experiment is shown in Appendix F. These extensions broaden the application of CLIPNeRF or other optimization- based editing methods to complex scenes.
Discussion, Limitations, and Conclusions
In this work, we propose distilled feature field (DFF), a novel method of NeRF scene decomposition for selective editing. We present quantitative evaluations of segmentation and extensive qualitative evaluations of editable novel view synthesis. In addition to these promising results, DFF-based models will benefit from future improvements self-supervised 21) foundation models. We also clarify future directions on editable view synthesis through our experiments, especially for smoothness priors and estimation of unobserved regions. Furthermore, while this work focuses editable view synthesis, it is also intriguing to transfer DFF to other applications, including 3D registration of text queries [16, 47, 4] robot teaching [32, 80].
The limitations of the DFF framework are two-fold. The first one is the upper bduround of performance due to distillation. The student model of distillation cannot largely outperform the teacher model'. If the resolution of teacher encoders low, the corresponding DFFs also becomes coarse-grained. If the LSeg cannot understand a text query, the -Seg-DFF also cannot. Secondly, the DFF uses volume rendering depending on the 3D reconstruction by NeRF. A NeRF model sometimes optimized to geometrically wrong solutions (e.g., floaters). Such geometry errors of the radiance fields would make supervision to DFFs noisy.
As possible negative societal impact, one might use our method for making realistic but fake content by editing NeRFs as desired. Automatic fake detection methods may help in preventing such misuse. NeRFs are further computation-intens. leading to high electricity usage. Recent work on efficient NeRFs [23, 57, 15] may alleviate this concern.
Acknowledgements
We thank Tsukasa Takagi, Toshiki Nakanishi, Hiroharu Kato, and Masaaki Fukuda for their helpful feedback.
References
[1) Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. ReferIt3D: Neural listeners for fine-gr tne-ggraied 3d object identification in real-world scenes. In 16th European Conference on Computer Vision (ECCV), 2020.
[2] Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (7):1425-1444 2016.
Because the official implementation is not available, implemented CLIPNeRF ourselves reproduce the paper to
the experiment Figure
our abilities. See appendix F for details. "While the DFF may outperform the teacher a little because the DFF denoises and improves the feature via multi-view fusion [112, 93], such improvement limited.
10

---

[3] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814, 2021.
[4] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[5] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Proceedings of the European Conference Computer Vision (ECCV), 2022.
[6] Sagie Benaim, Frederik Warburg, Peter Ebert Christensen, and Serge Belongic. Volumetric disentanglement for 3d scene manipulation. arXiv, 2022.
[7] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creet, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei- Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. arXiv, 2021. URL https: : /ttp/aaavvvv///////////////aaav/ org/ abs/2108 07258.
[8] Mark Boss, Raphael Braun, Varun Jampani, Jonathan Barron, Ce Liu, and Hendrik P. A. Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. URL http: //axiv..
org/ 2012 03918v4.
[9] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik Lensch. Neural-PIL: Neural pre-integrated lighting for reflectance decomposition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL pettps://operreeee
[10] Maxime Bucher, Tuan- Hunn-Hung VU, Matthieu Cord, and Patrick Pérez. Zero-shot se- mantic segmentation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 32, 2019. URL Pappee pdf.
www.eeeinnn....riiss.oom nwwninen.i.... cc/ patppe///19///1ll
[11] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1209-1218, 2018.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650-9660, October 2021.
[13] Rohan Chabra, Jan Eric Lenssen, Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d recon- struction. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. URL http://arxiv org/ abs 0bss/200 2003 10983v3.
[14] Anpei Chen, Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. URL
attp://arxis org/ abs/2103 15595v2.
[15] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Proceedings of the European Conference Computer Vision (ECCV), 2022.
11

---

[16] Dave Zhenyu Chen, Angel X Chang. and Matthias NieBner. Scanrefer: 3d object localization in rgb-d scans using natural language. 16th European Conference on Computer Vision (ECCV). 2020.
[17] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- supf-suue vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9640-966 9440-9649, 2021.
[18] Christopher Choy, Jun Yon'oung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 3075-3084 2019.
[19] Marius Cordts, Mohamed Omran, Sebastian Ramos. Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[20] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[21] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Moham- mad Norouzi, and Andrea Tagliasacchi. Nasa: Neural articulated shape approximation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. URL http://arxiv. org/ abs//91/// 03207v4.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openzrviiet net/ wow.ooodddididddnado
[23] Sara Kridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[24] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aure- lio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran As- sociates, Inc., 2013. URL ptttpe///proceedingg www.ig....u.s..... cc/ wwtpiiip..cppppppppppe pccpppppp//211//lleee.e (ic6Sc01007447777779777773333333
pdf.
[25] Xiao Fu, Shangzhan Zhang. Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. arxiv, 2022. URL https:// //tp:v////xvvve// org/ httg/abs/200 abs/2203.... 15224.
[26) Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. arXiv preprint trXiv:2105.06466 2021. URL
org/ httg/abb/2100..0em 06468v1.
[27] Jonathan Granskog, Till N Schnabel, Fabrice Rousselle, and Jan Novák. Neural scene graph rendering. ACM Transactions on Graphics (TOG), 40(4):1-11, 2021.
[28] Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. arXiv preprint wtXiv:2012.08503, 2020. URL http://arxiv wwwpswwwwww...r.eoooooooeo /2022 08503v1.
[29] Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3(12):4338-4364 2021. doi: 10. TP.1009/ 2000.3005434......
[30] Huy Ha and Shuran Song. Semantic abstraction: Open-world 3D scene understanding from
2D vision-language models. In Proceedings of the 2022 Conference on Robot Learning, 2022.
12

---

[31] Zekun Hao, Arun Mallya, Serge Belongie, and Ming- Yu Liu. Gancraft: Unsupervised 3d neural rendering of minecraft worlds. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. Utt. ptattp://arivv arriiv org/ abs/2104 07665v1..
[32] Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, and Jethro Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In Proceedings of International Conference on Robotics and Automation, 2018.
[33] Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang. Wei Sun, Yaonan Wang, Qiang Fu, Yanmei Zou, and Ajmal Mian. Deep learning based 3d segmentation: A survey, arxiv, 2021. URL httph:htt///// arxiv. org/ abs aa////////// 2103. 05423.
[34] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015. URL http://arxiv. org/ /503. /as./11530021... 02531.
[35] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5885-5894, October 2021.
[36] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL https: attps://arxix> org/ abs/2112. 01455.
[37] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. URL http: / /aaxiv.vvvv./////// org/ abo9 01750v1.
[38] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG), 2021. URL http:///rxivy org/ abs/ 2109. 11418v1.
[39] Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey, arxiv, 2020. URL org/ abs/b006..... 12057.
[40] Amit Kohli, Vincent Sitzmann, and Gordon Wetzstein. Semantic implicit neural scene repre- sentations with semi-supervised training. In International Conference on 3D Vision (3DV). IEEE, 2020. URL http:://at //trxiva.ivv//// org/ aaaaabss/2003 12673v2.
[41] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation. In CVPR, 2022.
[42] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
[43] Verica Lazova, Vladimir Guzov, Kyle Olszewski. Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. arXiv preprint 1rX1002204.10850, 2022.
[44] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language- driven semantic segmentation. International Conference on Learning Representations, 2022.
URL
[45] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3d video synthesis. arXiv preprint arXiv:2103.005597 2021. URL. http: ptett:///axivv wwwrrii.oorg//bbee org/abs/
2103. 02597v1.
13

---

[46] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In European conference on computer vision, pages 740-755. Springer, 2014.
[47] Haolin Liu, Anran Lin, Xiaoguang Han, Lei Yang. Yizhou Yu, and Shuguang Cui. Refer-it-in- rgbd: A bottom-up approach for 3d visual grounding in rgbd images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6032-6041) 2021.
[48] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. httg/bb//2000 11571v2.
URL hatt:://axxi
[49] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell, Editing conditional radiance fields. arXiv preprint 1xXiv:2105.06466, 2021. URL http:///axxxaa
org/ abs/2105. 06466v2.
[50] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer V2: scaling up capacity and resolution. arxiv, abs/2111 1.09883, 2021. URL https: / /twxxv.ivvv./////////a///aaaa orrrxiio.r////e (erggaba/2111 09883.
[51] Sebastian Loeschcke, Serge Belongie, and Sagie Benaim. Text-driven stylization of video objects. arXiv, 2022. URL https: //ax/v//avvvvaa http//awww.....aaaaaa//aa/
12396.
[52] Timo Lüddeeke and Alexander Ecker. Prompt-based multi-modal image segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), httpttps:: arrxiv. wwrxii..or//aai orwooogg/a....oooo..o httgg/bs/21112/ 2102.10003.
[53] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, URL attp://arxia orgo aaaaaaass/// 03828v2.
[54] Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In 2021 International Conference on 3D) Vision (3DV), pages 992-1002, 2021. doi: 10.1109/3DV53792 000079922222100111111
[55] Ben Mildenball, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
[56] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. URL http: / //arxiv. org/ abs/2003 08934v2.
[57] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 1121::102:1110::55 July 2022. doi: (0.1144//22222333533000 URL https://doi. org/10. 1145/3528223. 3530127.
[58] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990-4999, 2017.
[59] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. In Proc. ICCV, pages 7588-7597, 2019.
[60] Thu H Nguyen- MguueenPPuuce, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Block- gan: Learning 3d object-aware scene representations from unlabelled images. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6767-6778, 2020. URL https: htppppeeeeenng..ee/eeeeepeee neurips. cc/ httpa//www.....ooo.eeeeeeaeaaaaa
.df.
14

---

[61] Michael Niemeyer and Andreas Geiger. Giraffe:
Representing scenes as compositional gener- ative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puitern Recognition (CVPR), 2021. URL. http: / http/aixvvvv.///// org/ abs/2011 0tts/2011.220002.....11 12100v2.
[62] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), 2022.
[63] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. URL http:/ / /ttpivaaavvv///a///a///a./aaaaa/ .rgg
03110v2.
[64] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2856-2865, June 2021.
[65] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019. URL http: / hatp/aivvvvooo/////a///aaa www.aiwww//w...aa............eee abs 19101990.05550 05103v1.
[66] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensionn rep- resentation for topologically varying neural radiance fields, arXiv preprint arXiv:2106.13222 2021. URL http: /ttp:://aaxiii http///xxxvvv.r//e//ee abs/2106. 13228v2.
[67] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[68] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022. URL httpp:://aaat http////xvvvvv.////////////aa .rgg wtrggaab//229.... 14988.
[69] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 7748-8764 PMLR, 18-24 Jul 2021. URL https: htttps://proccedings. mlr, prwss/v139///eee http//a11//raarr/ee// ord21a. html.
[70] Sameera Ramasinghe, Lachlan E. MacDonald, and Simon Lucey. On regularizing coordinate- mlps. arxiv, 2022.
[71] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense predic- tion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
org/ abs.//03./2 13413.
[72] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language gugdeed dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[73] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2021. URL http://arxiv. http://arxii.oogga///0/010// 12490v1.
[74] Nils Reimers and Iryna Gurevych. Sentence-BERY Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural
15

---

Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 3982-3992, November 2019. doi: 10.19453.
UBRL.https://aclatthooll
org/D19-14111
[75] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, November 2020. doi: 10.18653/ v1/2020.emmll-maing main. URL https: hatp:p::/accaanthhioi
.rg. emnlp- main. 365.
[76] Zhongzheng Ren, Aseem Agarwala Bryan Russellt, Alexander G. Schwing?, and Oliver Wang". Neural volumetric object selection. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. ( alphabetic ordering).
[77] Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[78] David Rozenberszki, Or Litany, and Angela Dai. Language-groundee indoor 3d semantic segmentation the wild. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[79] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. arXiv, 2022. URL https: // //arxiv. org/ barg//bb//an aa/////bb//2...//// 11487.
[80] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se(()-equivariant object representations for manipulation. arXiv preprint arXiv:2112.05125: 2021.
[81] Vincent Sitzmann, Justus Thics, Felix Heide, Matthias NicBner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2019.
[82] Vincent Sitzmann, Michael Zollhofer, and Gordon Wetzstein. Scene representation net- works: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2019. URL http:
//arxiv. abs/ 190..996600 01618v2.
[83] Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B. Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv, 2022.
[84] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene under- standing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[85] Karl Stelzner, Kristian Kersting, and Adam R. Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arxiv, 2021. URL http:// org/ aa//a//s///00 01148.
[86] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arxiv, 2019.
[87] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempit- sky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2159-2159, January 2022, URL https:/ arrxivv. org/ abs 221.2.000111
16

---

[88] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geomet- ric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. URL http:// /ttp://arxiv.. aawii.xiiivi..... org/ abs/aaaa...... 10994v1.
[89] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable large scene neural view synthesis. arXiv, 2022.
[90] A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, Z. Xu, T. Simon, M. Nießner, E. Tretschk, L. Liu, B. Mildenhall, P. Srinivasan, R. Pandey, S. Orts-Escolano, S. Fanello, M. Guo, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, D. Goldman, and M. Zollhöfer. Advances in neural rendering. In ACM SIGGRAPH 202/ Courses, SIGGRAPH '21, 2021. doi: 10. 1445/445500.34445777777 URL https: //ap:////rxxvvv org/ wttps//11...00 05849..
[91] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of the International Conference Machine Learning (ICML), pages 10347- 10357, 2021.
[92] Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. arXiv preprint (rXiv:21 2021.
[93] Vadim Tschernezki. Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields:
3D distillation of self-supervieed 2D image representations. In Proceedings of the International Conference on 3D Vision (3DV), 2022. URL https::/ //arii//////////// http/dd//2200 2209003494 pdf.
[94] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construc- tion of large-scale nerfs for virtual fly-throughs. arxiv, 2021. URL http::ts:///rrt org/ abs/2112 221.2212...2 10703.
[95] Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and CV Jawahar. Idd: A dataset for exploring problems of autonomous navigation in unconstrained environments. In 2019 IEEE Winter Conference on Applications of Computer Vision, pages 1743-1751. IEEE, 2019.
[96] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL
wwwp:iinnnnnnuur..meeemm www.neee....ppaaaeeee mear.ips.cc/ppppr///////.... pptp//////////// pdf.
[97] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[98] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAT). International Joint Conferences on Artificial Intelligence Organization, 2021. URL http: //arxiv. org/ www.///100 /2066. 10689v1..
[99] Zhaoqing Wang. Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL https://arxie www./or//////aiooooaee/e/oo abs//aab....aaa/////a..aaaaaa1/a 15174.
[100] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi- niew stereo. Sep 2021. URL.
http://arxiv. org/ abs/2109, 01119..0112991. 01129v2.
17

---

[101] Xiaoshi Wu, Hadar Averbuch-Elow Jin Sun, and Noah Snavely. Towers of babel: Combining images, language. and 3d geometry for learning multimodal vision. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 428 437, October 2021.
[102] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond, 2021. URL httph:ttt/eeeaaletteeeeee/
cs www.brrww.ooo................... edu/.
[103] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. arxiv, 2021. URL https: /ttp::///axvvvvii org. abs 2/////11...1.. 14757.
[104) Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-comp coneectcompoiiiional neural radiance field for editable scene rendering. International Conference on Computer Vision (ICCV), October 2021. URL butps://arxia wwt//rg///////eee//oo//ee abtaaabb//110 2109010177..
[105] Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in room: Amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects. ACM Trans. Graph,, 11(4):0001111100:00 July 2022. doi: 00.1145/352222335531666 URL httpp:///dae org/ 111./1522/2222. 3530163.
[106] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2020. URL http: / arxiv org/ ab03 09852v3.
[107] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. Proc. ICCV, 2021.
[108] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multi- task learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[109] Hong- Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. International Conference on Learning Representations, 2022. URL https:// openreview. www.rreieww.eeeffreeeeee
[110] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. Ners: Neu- ral reflectance surfaces for sparse-view 3d reconstruction in the wild. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29835-29847. Curran As- sociates, Inc., 2021. URL BBeecaaeB44000533313333322222222
wttpa/ps.c.pppppr/e///0.... wcpppppe////2////1eeeee pdf.
[111] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. Nerfactor: Neural factorization of shape and reflectance under unknown illumination. arXiv preprint arXiv:2106.01970, 2021. URL http: happ://aaxxx /arxiviiivvv.//////aaaa/aaaaavla www.iii.orrg..o orggabs/2105 01970v1.
[112] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. URL http: org/ abs/ 15875v5.
[113] Shuaifeng Zhi, Edgar Suear, Andre Mouton, lain Haughton, Tristan Laidlow, and Andrew J. Davison. iLabel: Interactive neural scene labelling. arXiv, 2021.
18

---

[114] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai. Lu Yuan, Yin Li, and Jianfeng Gao. Regionelip: Region-based based language-image pretraining. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022. URL. https://p//// httitips//aaxvi...e/ org/ awsaaasss/111 09106.
[115] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3))302-321, 2019.
19

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The claims are empirically demonstrated in Section 5, and described in the whole paper.
(b) Did you describe the limitations of your work? [Yes] See mainly Section 5, and 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Mathematical formulations of the models are written in Section 3 and 4.
(b) Did you include complete proofs of all theoretical results? [N/A] Mathematical formu- lations of the models are written in Section and
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The complete code for the reproduction of all the experimental results is not publicly available. Be- cause the code is modification from public code by Zhi et al. [112], reproduction is also easier than from scratch. We will make our scene dataset publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Some of them are further described in the supplementary material.
(c) Did you report error bars (c.g.. with respect the random seed after running experi- ments multiple times)? [No] Error bars are not reported because would computa- tionally expensive and results expected be stable. Note that most existing studies on NeRF have not reported the bars too.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] It is difficult completely track and sum the total amount of computing in the experiments. Instead, we reported the setup of the main experiments.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Codebase and datasets are appropriately cited, mainly in Section 5.
(b) Did you mention the license of the assets? [No] We refer the readers to the original source instead of mentioning them in this paper.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include generated video by models in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A] No data about people.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] No such data.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] No such data experiment.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] No such data experiment.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No such data or experiment.
20


Reviews:
[{"review_id": "rVs6u0h7if-", "reviewer": "Reviewer_cXkV", "summary": "The paper studied an important problem, i.e., editing NeRF. It argued that previous works show difficult to selectively edit specific regions or objects. Thus, the paper decomposed scene semantics by learning a semantic feature descriptor distilled from the pretrained CLIP-LSeg or DINO models. The full method DFF (Distilled Feature Field) supports simple query-based decomposition and editing.", "questions": "Overall, the presented method is interesting and novel. But there are some unclear points w.r.t. the implementation details and experiments. The reviewer hopes the authors can answer the questions in the \"Weaknesses\" part.\n\n\n************************\nThanks for answering my questions. I am happy to raise my rating to \"Weak Accept\", and hope the authors can carefully addresses the concerns in the revision. Especially, please explain more on the implementation details and clearly present the editing details.", "limitations": "Possible limitations have not been discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. It is smart to exploit the distilling avenue to allow query-based decomposition and editing for NeRF.\n\n2. The DFF framework is well presented in Figure 1 (left). And the method part clearly explains how to learn a semantic feature descriptor by taking the pretrained CLIP-LSeg or DINO models as teachers.\n\n3. The paper examines different editing operations and shows many qualitative results to demonstrate the effectiveness of DFF.\n\n\n**Weaknesses**\n\nWhile the DFF approach is really interesting, the reviewer has some concerns w.r.t. the implementation details and experiments. \n\n1. The reviewer might miss some details, but it seems the paper has not clearly presented how to perform scene editing after training a DFF. Do we need to individually train a target scene for each editing operation? \n\n2. As the paper has stated that the code of complete reproduction of all the results is not yet publicly available, it would be better to include the training details in the main paper.\n\n3. Is it possible to make some quantitate comparisons with other NeRF editing approaches?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WixSHEi4EsX", "reviewer": "Reviewer_XqPY", "summary": "To edit a scene more easily, the authors propose a semantic scene decomposition of NeRF by leveraging distillation techniques to distill the 2D feature extractor such as LSeg or DINO into a 3D feature field. Therefore, the distilled NeRF can segment the regions of interest with given texts.", "questions": "1. Overall, the paper introduces a distillation method for NeRF semantic segmentation. Although it works well on several scenes, it is highly suggested to test your method on other indoor scenes of LLFF dataset and compared with SOTA indoor 3D semantic segmentation dataset (e.g. Scannet), as you already adopted a highly complicated 2D segmentation model.\n2. The authors verify their method based on vanilla NeRF. How about the generalization ability on other NeRF variants? For example, MipNeRF, NeRF for unbounded scenes? You can try to integrate the distillation into an unbounded NeRF variant and show the segmentation quality/view consistency effect by quantitative metrics and visualized videos (e.g. on KITTI dataset with some SOTA segmentation models).", "limitations": "See the weakness and questions.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is easy to understand and follow.\n2. The edited results on scene \"Flower\" looks fancy.\n\nWeaknesses:\n1. The claim is not rigorous. The authors claim \"feature encoders pre-trained in a self-supervised framework on the image domain\". However, this is partly true as LSeg requires semantic segmentation annotation for training.\n2. As the author argues \"We note that the goal of this paper is not to achieve state-of-the-art performance on 3D semantic segmentation tasks.\". However, this will weaken your contribution as you already learned from a large-scale pre-trained model (LSeg), but do not plan to achieve SOTA performance even on NeRF-related datasets. Further, you compare only one 3D segmentation method in Table 1, which contradicts your claim.\n3. Mixed-use of Figure and Fig.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "D96WSsH2zgR", "reviewer": "Reviewer_48WN", "summary": "The authors propose a novel approach for getting a NeRF to output both (i) the standard color, and, novel for this work (ii) zero shot-trained features that can be used for semantic segmentation / decomposition / editing. Put differently, whereas previous work had parallel branches for generating, eg, closed-set semantic segmentation labels, this work generates features that can be used with standard zero-shot segmentation techniques. These features are trained via a distillation using LSeg / DINO-based teacher networks.", "questions": "Would the approach be able to train just the feature generation branch? While that would not be very useful in the current context, it would nevertheless be interesting to know if the image branch is absolutely necessary for training the feature branch.", "limitations": "It would be useful to have a discussion on potential failures.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positive:\n- The paper is generally well written and easy to understand.\n- The topic is both very relevant and novel -- there is much work on the geometric side of NeRF, but much less on semantics and even less on feature generation and semantics / editing.\n- The approach is principled.\n\nNegative:\n- While the feature generation parts is fairly well explored, I found the actual editing part to not be explained very much /  very well. Unless I'm missing something, most of the text focuses on the generation of the features and the decomposition, but the editing is captured only by \"We can combine this with more complex edits, including optimization-based methods like CLIPNeRF\". I would very want more detail to be added here.\n- While the overall structure of the proposed approach is novel (to my knowledge), the individual components are not. I would almost go as far as to say that I see this as a systems paper.\n\nOverall I am quite positive about the paper -- (i) the methodology, while simple, is clear and appears novel; (ii) the results are very good. If accepted, I think the work might end up being quite influential.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "80yT57QNMbf", "reviewer": "Reviewer_1TRr", "summary": "This paper presents a simple yet novel solution to achieve dense volumetric semantic segmentation on NeRF, which supports semantics-guided NeRF decomposition and local editing. This decomposition is achieved by adding an additional renderer that renders the volumetric feature field similarly to radiance by distilling from a pre-trained 2D image semantic feature encoder like CLIP-LSeg. A rich set of experiments is presented to show the decomposition quality, generalization capability to different text and even image queries, and local editing (appearance and geometry) results based on the decomposition.", "questions": "I think most of the technical questions and evaluations I would like to see are addressed in the paper. Some minor questions I have:\n1. The difference between image segmentation and NeRF segmentation is that image pixels are equally weighted but volumetric sample points have naturally-defined weights (density). Would this cause uncertain/noisy labels in less certain regions (empty space or inside the objects)? I know these regions might not be as important as object surfaces, but just being curious.\n2. Is the algorithm robust against artifacts that are commonly seen in real-world NeRF (like misalignment and cloudy artifacts) due to less-perfect camera or input image artifacts?\n3. The distillation supervision is applied on alpha-blended renderings, would this be a potential issue that the training tries to overfit the 2D feature supervision with bad volumetric features on the samples? Especially on NeRF trained on narrow baselines? The radiance may have a similar issue but the difference is that the ultimate results for radiance are still rendered views, but here we want to directly use the volumetric features.\n4. Is there any boundary issues with the segmentation? Like incomplete regions or holes/noisy points?", "limitations": "There is not enough discussion on the limitations as far as I can find. I would suggest the authors add such discussions in the next version.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ The major strength of this work is that it allows high-quality volumetric NeRF semantic segmentation by reusing the zero-shot image segmentation models.\n\n++ The method is simple and easy to implement, allowing for good results on various applications and works on various NeRF scenes without needing extra training data.\n\n++ Solid experiments and promising results, quantitatively and qualitatively.\n\n-- The method does not show any significant technical breakthrough and is heavily based on LSeg, which limits its technical depth. But I do not think it is a major issue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept