NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes
Authors: Zhiwen Fan
Peihao Wang
Yifan Jiang
Xinyu Gong
Dejia Xu
Zhangyang Wang
Venue: ICLR 2023 poster (2023)
Keywords: neural radiance field
self-supervised learning
object segmentation

Abstract: Neural volumetric representations have shown the potential that Multi-layer Perceptrons (MLPs) can be optimized with multi-view calibrated images to represent scene geometry and appearance without explicit 3D supervision. Object segmentation can enrich many downstream applications based on the learned radiance field. However, introducing hand-crafted segmentation to define regions of interest in a complex real-world scene is non-trivial and expensive as it acquires per view annotation. This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object Segmentation (NeRF-SOS), couples object segmentation and neural radiance field to segment objects in any view within a scene. By proposing a novel collaborative contrastive loss in both appearance and geometry levels, NeRF-SOS encourages NeRF models to distill compact geometry-aware segmentation clusters from their density fields and the self-supervised pre-trained 2D visual features. The self-supervised object segmentation framework can be applied to various NeRF models that both lead to photo-realistic rendering results and convincing segmentation maps for both indoor and outdoor scenarios. Extensive results on the LLFF, BlendedMVS, CO3Dv2, and Tank & Temples datasets validate the effectiveness of NeRF-SOS. It consistently surpasses other 2D-based self-supervised baselines and predicts finer object masks than existing supervised counterparts.

Published as a conference paper at ICLR 2023
NERF-SOS: ANY-VIEW SELF-SUPERVISED OBJECT SEGMENTATION ON COMPLEX SCENES
Zhiwen Fan', Peihao Wang', Yifan Jiang', Xinyu Gong', Dejia Xu', Zhangyang Wang' Department of Electrical and Computer Engineering, University of Texas at Austin (zhi fan at w edu
ABSTRACT
Neural volumetric representations have shown the potential that Multi-layer Per- ceptrons (MLPs) can be optimized with multi-view calibrated images to represent scene geometry and appearance without explicit 3D supervision. Object segmenta- tion can enrich many downstream applications based on the learned radiance field. However, introducing hand-crafted segmentation to define regions interest in a complex real-world scene is non-trivial and expensive it acquires per view anno- tation. This paper carries out the exploration of self seff-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object Segmentation (NeRF-SOS), couples object seg- mentation and neural radiance field segment objects in any view within a scene. By proposing a novel collaborative contrastive loss in both appearance and geome- try levels, NeRF-SOS encourages NeRF models to distill compact geometry-awarr segmentation clusters from their density fields and the self-supervised pre-trained 2D visual features. The self-supervised object segmentation framework can be ap- plied to various NeRF models that both lead to thals-realistic rendering results and convincing segmentation maps for both indoor and outdoor scenarios. Extensive results on the LLFF, BlendedMVS, CO3Dv2, and Tank & Temples datasets validate the effectiveness of NeRF-SOS. It consistently surpasses 2D-based self-supervised baselines and predicts finer object masks than existing supervised counterparts. Code is available at: https: / http///ttttbbttt// com wwm/VITT 1117A-rruup Group/NeRF--ae NeRF- SOS
INTRODUCTION
Scene modeling and representation are essential to the computer vision community. For instance, portable Augmented Reality (AR) devices such as the Magic Leap One reconstruct the scene geometry and localize users (DeChicchis) 2020). but they often struggle to comprehend the surround- ing objects. This limitation poses challenges when designing interactions between humans and the environment. Although human-annotated data from diverse environments could mitigate the hurdles of understanding and segmenting the surrounding objects, collecting such data often costly and iime-consuming. Therefore, there is growing interest in developing intelligent geometry modeling frameworks that can learn from unsupervised or self-supervised techniques.
Recently, neural volumetric rendering techniques, such as neural radiance field (NeRF) and its variants (Mildenhall et al., 2020al Zhang et al., 2020 Barron et al., (021), have demonstrated exceptional performance in scene reconstruction, utilizing multi-layer perceptrons (MLPs) and calibrated multi-view images to generate fine- -reingr unseen views. While several recent works have explored scene understanding with these techniques (Vora et al. [021] Yang et al. 2021 Zhi cet al., 202T), they often require either dense view annotations to train a heavy 3D backbone for capturing semantic representations (Vora et al. 2021 Yang et al. 2021), provide sparse semantic labels (Zhi et al. [202]) Although recent self self-super object discovery approaches neural radiance fields (Yu al. 2021c; Stelzner et al. (021 have been effective in decomposing objects on synthetic indoor data, there still a sigmificant gap to be filled in applying these approaches complex real-world scenarios.
human intervention to

---

Published as a conference paper at ICLR 2023
Color Images
DINO-CoSeg
SemanticNeRI
Visual examples. From left to right: ground truth color images, annotated object masks, object masks rendered by NeRF-SOS, 2D image co-segmentation using DINO (Amir al. 2021), and object masks rendered by Semantic-NeRF (Zhi et al. 2021), respectively. NeRF-SOS outperforms the previous methods by generating object masks with precise local details.
Figure 1:
In contrast to previous works, we investigate more generic setting, by using general NeRF models to segment 3D objects in real-world scenes. We propose new self-supervised object segmentation framework for NeRF that utilizes collaborative contrastive loss. Our approach combines features from a self-supervised pre-trained 2D backbone ("appearance level") with knowledge distilled from the geometry cues of scene, using the density field of NeRF representations ("geometry level"). To be more specific, we adopt supersupe approach to learn from a pre-trained 2D feature extractor, such DINO-ViT (Caron et al. 2021) and incorporate the inter-view visual correlations generate distinct segmentation feature clusters within the NeRF framework. We introduce a geometry-level contrastive loss by formulating geometric correlation volume between NeRF's density field and the segmentation clusters to make the learned feature clusters aware of scene geometry. Our proposed self-supervised object segmentation framework tailored for NeRF, dubbed NeRF-SOS, serves as general implicit framework and can be applied to any existing NeRF models with end-to-end training. We implement and evaluate NeRF-SOS, using vanilla NeRF (Mildenhall al. | (020a) for real- world forward-fan -arrirdfaiingggaoo datasetss (LLFF (Mildenhall et al. 2019)), object-centric datasets (BlendedMVS YYo cct al. 2020) and CO3Dv2 (Reizenstein et al. 2021)): and using NeRF++ (Zhang et al. 2020) for outdoor unbounded dataset (Tank and Temples (Riegler & Koltun 2020)). Experiments show that
NeRF-SOS significantly outperforms existing object discovery methods and produces ciew-consistent segmentation clusters: few examples shown in Figure 1
We summarize the main contributions as follows:
We explore how to effectively apply the self-supervised learned 2D visual feature for 3D repre- sentations through an appearance contrastive loss, which forms compact feature clusters to allow any-view object segmentation in complex real-world scenes.
We propose new geometry contrastive loss for object segmentation. By leveraging its density field, our proposed framework injects scene geometry into the segmentation field, making the learned segmentation clusters geometry-aware.
The proposed collaborative contrastive framework can be implemented upon NeRF and NeRF++, for object-centric, indoor, and unbounded real-world scenarios. Experiments show that our self-supervised object segmentation quality consistently surpasses 2D object discovery methods and even yields finer segmentation results than the supervised NeRF counterpart (Zhi et al.) 2021).
RELATED WORK
Neural Radiance Fields NeRF is first proposed by Mildenhall et al. Millennall et al. (020b). which models the underlying 3D scenes continuous volumetric fields of color and density via layers of MLP. The input NeRF is a 5D vector, containing a 3D location (z. y, 2) and 2D viewing direction (0, Φ). Several following works emerge trying to address its limitations and improve
2

---

Published as a conference paper at ICLR 2023
the performance, such unbounded scenes training (Zhang al. 2020 B Barron al. 2021 fast
training (Sun et al., [002]] | Deng et al. (02T), efficient inference (Rebain et al. 20200] Liu et al.
2020
Lindell et al.] 22200] Garbin [202] Reiser al., 202I; Yu et al., 202
Lombardi et al. 2021).
better generalization (Schwarz. et al., 2020a| Trevithick & Yang 2020;
Wang et al. 2021h 1
Chan
et al.
2020 Yu et al. 2021b Johari et al.] [221]] Varma et al. 2022). supporting unconstrained
scene MMrtin-Bruall et al. 20en
Chen et al. [021] [u et al. 2022)) editing (Liu et al. 2021 Jiakai
ct al.
2021 Wang et al. [2021a] Jang & Agapito 2021 Kundu et al. 2022 Fan al. 2022). multi-task
learning (Zhi et al., ). In this paper, we treat NeRF as a powerful implicit scene representation
and study how to segment objects from complex real-world scene without any supervision.
Object Co-segmentation without Explicit Learning Our work aims discover and segment visually similar objects in the radiance field and render novel views with object masks. It is close to the object co-segmentation (Rother et al. 2006) which aims segment the common objects from 2018). Object so-semmenttiion has been widely adopted in computer vision set of images (Li et al.
and computer graphics applications, including browsing in photo collections (Rother et al.) 2006). 3D reconstruction (Kowdle et al. 2010), semantic segmentation (Shen et al. (017)) interactive image segmentation (Rother et al.] 2006), object-based image retrieval (Vicente et al. 201 and video object vacking/segmentttion (Rother et al.) 2006). The authors in (Rother et al. 2006 first shows that segmenting two images outperforms the independent counterpart. This idea is analogous to the contrastive learning way in later approaches. Especially, the authors in (Hénaff et al., 2022)
propose the self-supervised segmentation framework using object discovery networks. (Siméoni et al. 2021) localizes the objects with a self- self-super transformer. The paper (Hamilton et al. 2022)
introduces the feature correspondences that distinguish between different classes. Most recently, a new co-segmentation framework based DINO feature (Amir et al.) (02T) has been proposed and achieves better results on object co-segmentation and part co-segmentation.
However, extending 2D object discovery to NeRF is non-trivial as they cannot learn the geometric cues in multi-view images. uORF Yu et al. 2021c) and ObSuRF (Stelzner et al. (02T)) use slot-based CNN encoders and object-centric latent codes for unsupervised 3D scene decomposition. COLF (Smith al. (022) proposes a light field compositor module to accelerate NeRF--based object decomposition. Although they enable unsupervised 3D scene segmentation and novel view synthesis, experiments are on synthetic datasets with pre-defined categories, leaving gap for complex real-world applications. NVÓS (Ren et al. 2022) leverages users' scribbles for weakly-supervised object segmentation. The concurrent work, RFP (Liu et al. 2022 enables label-free object segmentation in real-world scenes with a propagation strategy. Panoptic NeRF Field (Kundu et al.) 2022) proposes to represent each object instance using a separate MLP with supervisions from other models. N3F (Tschernezki et al., 2022) minimizes the distance between NeRF's rendered feature and 2D feature for scene editing. Most recently, DFFs (Kobayashi et al. 2022> propose to distill the visual feature from supervised CLIP-LSeg or sulf-super DINO into 3D feature field via an element-wise feature distance loss function. can discover the object using a query text prompt or patch. In contrast, we design new collaborative contrastive loss on both appearance and geometry levels to find the objects with a similar appearance and location without any annotations. The collaborative design general and can be plug-and-play to different NeRF models.
3 METHOD
Overview This paper presents extension to existing NeRF models to enable object segmentation. As shown in Figure we augment NeRF models by appending a parallel segmentation branch to predict point-wise implicit segmentation features. Specifically, NeRF-SOS can render depth (o)) segmentation (8), and color (c). We then use a self-supervised pre-trained framework (such as DINO-ViT (Caron et al. 2021) generate feature tensor (f) from the rendered color patch (c), constructing an perearannc-segmentatie
correlation volume between f and s. Similarly, we instan- correlation volume using o and s. By generating positive/negative pairs from different views, we can distill the correlation patterns in both the visual feature and scene geometry into the compact segmentation field s. During inference, we use clustering operation (such as K-means) to generate object masks based on the rendered feature field.
tiate a woommtry--sementtaaee
3.1 PRELIMINARIES
Neural Radiance Fields NeRF (Mildenhall et al. 2020a) represents 3D scenes as radiance fields via several layer MIPs,, where each point has a value of color and density. Such radiance field can
3

---

Published as a conference paper at ICLR 2023
geomeetry-segmeenttioo
Density:
Viewing Erstum
Ronderod
G.y.s. p,p)
Gcomatry
segmentziool
Kendere
Seynestation:
cverchutio
Gen-segg
MID
welme
appeurance Seeeaarance-seemmaattin
Appecanace Rennerod
srgmertation
Colure
correlation
DINO
Rendered
The overall pipeline of the proposed NeRF-SOS. Input with rays cast from multiple views, we render the corresponding color patch (c), segmentation patch (s), and depth patch (o). Then, appearance-segmentalie formulate a collaborative contrastive loss, enabling NeRF-SOS to render object masks from any viewpoint using the distilled segmentation field.
Figure 2:
correlations and geometry-segmentt
correlations are used to
be formulated F (zx 0) + (c. where E RR is the spatial coordinate, 0 E [-T,, denotes the viewing direction, and с E R², E R+ represent the RGB color and density, respectively. To (o, d, 0) for each pixel the image plane, where R form image, NeRF traces ray TR3 the direction of the ray, and 0 E [-x, T] the angular denotes the position of the camera, d E viewing direction. Afterwards, NeRF evenly samples K points (tx {t,) f between the near-far bound
[fn, tyl along the ray. Then, NeRF adopts volumetric rendering and numerically evaluates the ray integration (Max 1995) by the quadrature rule:
C(r) Σ T(k((1 exp(-ords))es where T(k) exp (
Σ
where 6j fx+1 ls intervals between sampled points, and (ck, ok) F(o + tgd, 0) output from the neural network. With this forward model, NeRF optimizes the photometric loss between rendered ray colors and ground-truth pixel colors defined follows: Ephotometrie
E(. cc()) 2l where R defines a dataset collecting all pairs of ray and ground-truth colors from captured images.
3.2 CROSS VIEW APPEARANCE CORRESPONDENCE
Semantic Correspondence across Views Tremendous works have explored and demonstrated the importance of object appearance when generating compact feature correspondence across views (Hénaff al. 2022 Li et al. 2018). This peculiarity then utilized in self-supervised 2D semantic segmentation frameworks (Hénaff et al. 2022 Li et al. 2018 Chen et al.] (020) to generate semantic representations by selecting positive and negative pairs with either random or KNN-based rules (Hamilton et al. 2022). Drawing inspiration from these prior arts, we construct the visual feature correspondence for NeRF at the appearance using a heuristic rule. To be more specific, we leverage the self-supervised model (c.g., DINO- ViT (Caron et al. (02I)) learned from 2D image sets to distill the rich representations into compact and distinct segmentation clusters. four-layer MLP is appended to segment objects in the radiance field parallel to the density and appearance branches. During training, we first render multiple image patches from different viewpoints using Equation [ then feed each batch into DINO-ViT generate feature tensors of shape H/' x W' x C". They are then used to generate the appearance correspondence volume (Teed & Deng. 2020 Hamilton al. 2022) across views, measuring the similarity between regions of different views:
fehw felt |frwel 1fkrrrl
Fhaskt =
where / and /' stand for the extracted DINO feature from two random patches in different views, e is the feature dimension of DINO, (h, w) and (h'; w') denote the spatial information feature tensor for f and f', respectively, and the traverses through the feature channel dimension.
Distilling Semantic Correspondence into Segmentation Field The correspondence volume F from DINO has been verified it has the potential in unsupervised semantic segmentations (Hamilton
4

---

Published as a conference paper at ICLR 2023
We next explore how to learn segmentation field by leveraging F. Inspired by CRF and STEGO (Hamilton et al. 2022) where they refine the initial predictions using color or ceature-corrllate regions in the 2D image. We propose to append an extra segmentation branch to predict the segmentation field, formulating segmentation correspondence volume by leveraging its predicted segmentation logits using the same rule with Equation 2 Then, cenearancc-seementttie correlation aims to enforce the elements of s and s' closer if f and f' are tightly coupled, where the expression with and without the superscripl indicates two different views. The volume correlation can be achieved via an element-wise multiplication between S and F, and thereby, we have the appearance contrastive loss Lappi
et al. 2022
construct the
Cupp (r, =
(Fhooh b)S Shawkww hank'an
Σ
(3)
Capp CapCapp (Fid, bia) Xneg Capp (Fnegs Uneg)
where Shak'w' -e Schoo
indicates the segmentation correspondence volume between two
views, is the cast ray fed into NeRF, is a hyper-parameter to control the positive and negative pressure. Aid and Xneg indicate loss force between identity pairs (positive) and distinct pairs (negative). The intuition behind the above equation is that minimizing Lapp with respect to S, to enforce entries in segmentation field to be large when F b are positive items and pushes entries to be small F - b are negative items.
Discover Patch Relationships To construct Equation 4 we build cosine similarity matrix to effectively discover the pos- itive/negative pairs of given patches. For each matrix, we take N randomly selected patches as inputs and adopt pre-trained DINO-VIT to extract meaningful representations. We use the [CLS] token from VIT architecture to represent the semantic features each patch and obtain positive pairs by the diago- nal entries and N negative pairs by querying the lowest score in each row. An example using three patches from different views shown in Figure 3 Similar to [Tumanyan et (2022). observe that the [CLS] token from a self-supervised pre-trained ViT backbone can capture high-level semantic appearances and can effectively discover similarities between patches during the proposed end-to-end optimization process.
1.0
0.83
0.71
0.83
1.0
0.68
0.71
0.68
1.0
Figure 3: Cosine similarity matrix calculated on scene Fortress.
3.3 CROSS VIEW GEOMETRY CORRESPONDENCE:
correlation" enables the clustered features with similar appearance together. However, appearance alone may cause spatial discontinuities, DINO-VIT may overfocus to capture semantic parts rather than making the clusters spatial smooth (see Figure 7 Therefore, propose geometric correlation volume penalize discontinuities between neighboring points, by formulating the attractiverrepuliiv force using point-wise distance.
Constructing the
Geometry Correspondence across Views Apart from distilling the visual feature from DINO into the segmentation field we propose to leverage the density field that already exists in NeRF models to formulate a new geometry contrastive loss to encourage spatial coherence, Specifically, given batch of M cast ray r as NeRF's input, we can obtain the density field of size M K where K indicates the number of sampled points along each ray. By accumulating the discrete bins along each ray, we can roughly represent the density field as a single 3D point:
== To + ra D
(5)
K D(r) Σ -xppo--=0p()) k=1
(6)
where p is the accumulated 3D point along the ray, D is the estimated depth value of the corresponding pixel index. Inspired by Point Transformer (Zhao et al., 202I) which uses point-wise distance as representation, utilize the estimated point position as geometry formulate geometry
5

---

Published as a conference paper at ICLR 2023
Table I: Quantitative comparison of the novel view synthesis and object segmentation of LLFF
dataset the scenes Flower and Fortress.
Scene "Flover"
PSNRT
SSIM
NV-AR1
loU/BG)
JoL(()))
0.2666
0:7123
0.4267
DOCS
0L099
0.4224
0.2461
0.3643
DINowrasay
0.9946
B:036
0.966
Samante -eRe
(Sugervised)
0.500
0.1739
0:743
Scone Fortiess
PSNR
SSIN
LPIPS
NV.ART
IFM Spnzeess a 2011
0.3700
0.7799
0.4536
0.663
DOCS
0.3412
09329
0.7265
0.8297
DINO+CaSiy Amirr & 2021
09503
09686
0.9395
0,9640
NeKH-SOS
019012
77h 2021
0.8576
Color Images Mask Annotations
Ours
DINO-CuSeg
DOCS
Sem.NeRF
Figure 4: Qualitative results on scene Flower and Fortress of LLFF dataset. In the fourth column, DINO-CoSeg mistakenly matches several discrete patches, as DINO has higher activation on just a few tokens, which may lead to view-inconsistent and disconnected co-segmentation results. superscript denotes the supervised method. DOCS and DINO-CoSeg are not able to perform novel view synthesis, and thus we perform rendering before segmentation using vanilla NeRF
level correspondence volume across views by measuring point-wer moint-wise absolute distance:
Ghwil wf
Σ Johow
+
where g and g' are the estimated 3D point positions in two random patches of different views, c is (h, w) and (h', (h',w") w') denote the spatial location on feature tensor for and g' respectively. Injecting Geometry Coherence into Segmentation Field To inject the geometry cue from the density field to the segmentation field, we formulate segmentation correspondence volume S and geometric correspondence volume G using the same rule of Equation 3 By pulling/pushing positive/negative pairs for the geometry-ssgmmentttion correlation of Equation a new geometry- aware contrastive loss Lyeo
come up with
(r. =
Σ (Gowh wi b)Shwhh'"
(8)
hwh'ww
Egeo XidCoeo(Tids bid) + AnegCgeo neg: bneg)
Same appearance contrastive loss, find positive pairs and negative pairs via the pair-wise cosine similarity of the [CLS] tokens.
3.4 OPTIMIZING WITH STRIDE RAY SAMPLING
We adopt patch-wise ray casting during the training process, while also leverage Stride Ray Sampling strategy, similar to prior works (Schwarz et al. 2020b Meng et al. 202T to handle GPU memory bottleneck. Overall, optimize the pipeline using a balanced loss function:
L Ap + d1 Copp + Agen
(10)
where Ao. Å1, and A are balancing weights.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets We evaluate all methods on four representative datasets: Local Light Field Fusion (LLFF)
dataset MMilennhall et al. 2019), BlendedMVS (Yao al. 2020), CO3Dv2 (Reizenstein et al. 2021

---

Published as a conference paper at ICLR 2023
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 5: Novel view object segmentation results on object-centric datasets: BlendedMVS (the 1st row) and C03Dv2 (the 2nd row). NeRF-SOS (the 3rd column) produces masks with finer details.
Color Images Mask Annolations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 6: Novel view object segmentation results on unbounded scene Truck. NeRF-SOS (the 3rd column) produces more view-consistent masks than other self-supervised methods. It even generates finer details than supervised ComaattccNeRF+ee (see the gaps between wooden slats the top row and the side view mirror in the bottom row).
and Tank and Temples (T&T) dataset (Riegler & Koltun 2020). Particularly, the forward- facing scenes {Flower, Fortress] from LLFF dataset, two object-centric scenes from BlendedMVS dataset, common objects (Backpack, Apple) captured by video sequences from CO3Dv2 dataset, and unbounded scene Truck from hand-held 360 capture large-scale Tank and Temples dataset. We choose these representative scenes because they contain at least one common object among most views. We manually labeled all views as a binary mask to provide fair comparison for all methods and used them to train Semantic-NeRF. Foreground objects appearing in most views are labeled as while others are labeled as O. We train and test all methods with the original image resolutions.
Training Details We first implement the collaborative contrastive loss upon the original NeRF (Mildenhall et al. (020a).. In training. we first train NeRF-SOS without segmentation branch following the NeRF training recipe (Mildenhall al., 2020b) for 150k iterations. Next, we load the weight and start to train the segmentation branch alone using the stride ray sampling for another 50k iterations. Model weights except the segmentation branch are kept frozen in the second phase. The loss weights Åo Åy. Ag. Aids and Aneg set 0, 1, 0.01, and in training the segmentation branch. The segmentation branch formulated as a four-layer MLP with ReLU as the activation function. The dimensions of hidden layers and the number of output layers set 256 and 2, respectively. The segmentation results are based on K-means clustering on the segmentation logits. We train Semantic-NeRF (Zhi al. (021) for 200k in total for fair comparisons. We randomly sample eight patches from different viewpoints (a.k.a batch size N 8) in training. The patch size of each sample
as 64 64, with the patch stride as We use the official pre-trained DINO- MITO-ViTOA in a self-supervised manner on ImageNet dataset as our 2D feature extractor. The pre-trained DINO backbone is kept frozen for all layers during training- All hyperparameters are carefully tuned by grid search, and the best configuration is applied to all experiments. All models are trained on an NVIDIA RTX A6000 GPU with 48 GB memory. We reconstruct N positives and N negatives pairs on the fly during training, given N rendered patches. More details can be found in the appendix.
Metrics We adopt the Adjusted Rand Index in novel views as a metric to evaluate the clustering quality, noted NV-ARI. We also adopt mean coarrectioonove-Unniee to measure segmentation quality for both object and background, as we set the clusters with larger activation as foreground by DINO. To evaluate the rendering quality, follow NeRF (Mildenhall et al., 2020a). adopting peak signal-to-noise ratio (PSNR), the structural similarity index measure (SSIM) [Wang et al.) (004). and learned perceptual image patch similarity (IPIPS) (Zhang et al.) 2018) evaluation metrics.
7

---

Published as a conference paper at ICLR 2023
Table 2: Quantitative evaluation of the novel view synthesis and object segmentation on BlendedMVS and CO3Dv2 datasets, with several 2D object discovery frameworks and the supervised Semantic- NeRF Results on each dataset are averaged on all scenes,
BlendedMVS
PSNR
SSIM
LPIPS
NV-ARI
IoU(BG)
[oU(FG) 1
mloU
IEM Savanese et al. 2021
0.1339
0,5615
0.3715
0.4665
DOCS
0:7031
0,7030
DINO+COsog Aiil & al.
2021
0.9074
0,9692
0.917
0,9431
NeRT-SOS (Ouns)
23,86
0.8089
0.1288
0.9280
0.9756
0.9347
0.9552
Semantic- NeRF (Zhi al. 2021
(Supervised)
23.84
0.8080
0.1339
0.9399
0.9805
0
0.9597
CO3Dv2
PSNR
SSIM
LPIPS J
NV-RI
loUTDG)
(oU(FG)
mloU
IBM Savarese et all 2021
0.4784
0,7983
0.5708
08645
DOC'S ar 2016
0.8918
0.9694
0.8928
0.9207
DINO+CoSeg al. 2021
0.8199
0.9559
0.8222
0.8891
NORF-SOS (Oirs)
30.37
0.9388
B.073
0.9381
0,9813
0.9401
0.9K47
Semantic-NoRF (Zhi at al. 2021
(Supervised)
31.17
0.9405
0.0603
0.9399
0.9821
0.9410
0.9615
Table Quantitative results of the object segmentation results on outdoor unbounded scene Truck, with several 2D object discovery frameworks and the supervised Semantic-NeRF.
Scene "Truct"
PSNK 1
SSIM
IPPIS
NV-ARI
loLi(H(3 1 loU((H()
mlot/ f
Savarese
0.6791
0.6395
DOCS ar 2013
0.1517
0.6845
0.2163
0:4654
DINO+CnSog pamir 2021
0.8571
0,9408
0.9080
0.9244
NoRF-SOS
22
0.7000
0.3691
0.9207
0.9689
0.9455
0.9572
Semantic-Neke
(Supervised)
21.08
0.6300
0.4114
0,9674
0.9782
4.2 COMPARISONS
supervied Object Segmentation on LLFF We build NeRF-SOS the vanilla NeRF (Mildenhall et al.) 2020a) to validate its effectiveness on LLFF datasets. Two groups of current object segmentation are adopted for comparisons: i. -NRF-
methods, including NeRF-SOS, and supervised Semantic-NeRt (Zhi et al. (021)) trained with annotated masks; id. image-based object co-segmentation methods: DINO-COSeg (Amir et al.) (02T)) and DOCS (Li et al.) 2018); and wi. single-image based unsupervised segmentation: IEM (Savarese et al. 2021) follows CIS (Yang al. 2019) minimize the mutual information of foreground and background. As image- based segmentation methods cannot generate novel views, we pre-render the new views using NeRF and construct image pairs between the first image in the test set with others for DINO-COSeg (Amir et al. 202T) and DOCS (Li al. 2018). Evaluations on IEM also use pre-rendered color images.
Quantitative comparisons against other segmentation methods are provided in Table I together with qualitative visualizations shown in Figure 4 These results convey several observations to us: 1). NeRF-SOS consistently outperforms image-based co-segmentation in evaluation metrics and view consistency. 2). Compared with SoTA supervised NeRF segmentation method (Semantic-NeRF (Zhi ct 2021)), our method effectively segments the object within the scene and performs on par in both evaluation metrics and visualization.
Self-supervised Object Segmentation on Object-centric Scenes For the object-centric datasets BlendedMVVS and CO3Dv2, we uniformly select 12.5% of total images for testing. CO3Dv2 provides coarse segmentation maps using PointRend (Kirillov al., 2020) while parts of the annotations are missing. Therefore, we manually create faithful binary masks for training the Semantic-Nele and evaluations. As we can see Table I Figure 5 our self- seff-supe NeRF method consistently surpasses other 2D methods. We deliver more details comparisons in our supplementary materials.
Self-supervised Object Segmentation on Unbounded Scene To test the generalization ability of the proposed collaborative contrastive loss, we implement it on NeRF++ (Zhang et al., 2020) to test with a more challenging unbounded scene. Here, we mainly evaluate all previously mentioned methods on scene Truck as it is the only scene captured surrounding an object provided by NeRF++. We re-implement Semantic-NeRF using NeRF++ as the backbone model for unbounded setting, termed Cemantic-NeRF+4 Compared with supervised Semantic-NeRF+++ NeRF-SOS achieves slightly worse results on quantitative metrics (see Table Yet from the visualizations, we see that NeRF-SOS yields quite decent segmentation quality. For example. 1). In the first row of Figure 6 NeRF-SOS can recognize the side view mirror adjacent to the truck. 2). In the second row of Figure 6 NeRF-SOS can distinguish the apertures between the wooden slats those apertures have distinct depths than the neighboring slats, thanks to the geometry-awane contrastive loss. Further, we show the 3-center clustering results on the distilled segmentation field in Figure
8

---

Published as a conference paper at ICLR 2023
Color Images (GT)
Seg. Masks
Sey. Masks (App.+Goo.)
Seg. Masks (Only App.)
Seg. Masks (Only Geo.)
Depth Maps (Estimated)
(Annotated)
Figure 7: Object segmentations using three loss variants are shown in columns 3, 4, and 5: the collaborative loss (AP.,+Geo.). appearance-only loss (App.); geometric-only loss (Geo.).
Color Image
Clusters
Clusters
ColorImage
Clusters
3 Clusters
Figure 8: Qualitative results on scene Truck with different cluster centers its distilled segmentation field. Note that the cross-view visualized colors of multiple-center clustering are not corresponding the subject ID, we perform unsupervised clustering.
Table 4: Experiments on multiple NeRF-SOS variants. We show the results of joint training of the NeRF and contrastive loss the first row, NeRF-SOS with ResNer50 as feature extractor in the second row, and our final model in the last row.
Scone "Flover
PSNR
SSIM 7
LPIPS |
NV.ARI loLi(B))
[oL((PE)
mlat. 7
NORF-SOS Joint trinnng)
16.96
0.4585
0.723
0.1961
0.7951
0.2220
0.5091
NcRF-SOS (ResNet)
0.8827
0.9124
NoRF-SOS (Two-stage training)
25.96
0.7717
0.1502
0.9529
0.9869
0.9503
0.9686
4.3 ABLATION STUDY
Impact of the Collaborative Contrastive Loss To study the effectiveness of the collaborative contrastive loss, we adopt two baseline models by only using appearance contrastive loss or geometric contrastive loss on NeRF++ backbone. As shown in Figure 7 we observe that the segmentation branch failed to cluster spatially continuous objects without geometric constraints (mloU: 0.5029). Similarly, without visual cues, the model lost the perception of the central object (mloU: 0.5516). Our full model constructs precise clusters with spatial coherence (mloU: 0.9689).
Joint Training with NeRF Optimization To demonstrate the advantages of two-stage training, we conduct an ablation study by jointly optimizing vanilla NeRF rendering loss and the proposed two-level collaborative contrastive loss. As shown Table 4 both the novel view synthesis quality and the segmentation quality significantly decreased when we optimize the two losses together. We conjecture the potential reason to be the fact that the optimization process of NeRF training affected by the confficting update directions, the reconstruction loss and the contrastive loss, which remains notorious challenge in the multi-task learning area (Yu al. 2020).
CNN-based Backbone for Feature Extraction DINO-ViT firstly concludes that VIT architecture can extract stronger semantic information than ConvNets when being self-supervised trained. To study its effect on discovering the semantic layout of scenes, apply self-supervised ResNet50 (He et al. 2020) as backbones. The results in the second row of Table imply that the ViT architecture is more suitable for our NeRF object segmentation in both expressiveness and pair-selection perspectives.
CONCLUSION, DISCUSSION OF LIMITATION
In this paper, we introduce NeRF-SOS, self-supervised framework that learns object segmentation for any view in complex real-world scenes. NeRF-SOS proposes collaborative contrastive loss in both the appearance and geometry levels. Comprehensive experiments are conducted on four different types of datasets with state-of-the-ar image-based object (co- (co))semmentation frameworks and fully supervised Semantic-NeRF. The results show that NeRF-SOS consistently outperforms image- based methods and sometimes generates finer segmentation details than its supervised counterparts. However, similar to other scene-specific NeRF methods, one limitation of NeRF-SOS is that cannot segment across scenes, which plan explore in future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We would like to express our gratitude to Xinhang Liu from HKUST and Zhongzheng Ren from UIUC for their invaluable contribution the experimental work presented in this paper. Their expertise and time proved to be immensely helpful in conducting the necessary comparisons with their methods, especially since their codes were not publicly available. We greatly appreciate their generosity and support throughout the research process.
REFERENCES
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features dense visual descriptors. arXiv preprint arXiv:2112.058144 2021.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruulla,, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Eric Chan, Marco Monteiro, Peter Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-
GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. hupp:://urziv.orr/axx//222222
2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on muchine learning, pp. 1597-1607. PMLR, 2020.
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Feng Ying, Xuan Wang, and Jue Wang. Hallucinated neural radiance fields in the wild, 2021.
Joseph DeChicchis. Semantic understanding for augmented reality and its applications. 2020.
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. arXiv preprint arXiv:2107.0279,, 2021.
Zhiwen Fan, Yifan Jiang, Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp- 636-654. Springer, 2022.
Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. https:/aarrivorrgaa22211....33 2021.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.0844 084144 2022.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9773 2020.
Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelovié. Object discovery and representation networks. arXiv preprint (iXi::2203.08777 2022.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2(7), 2015.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12949-12958, 2021.
10

---

Published as a conference paper at ICLR 2023
Zhang Jiakai, Liu Xinhang, Ye Xinyi, Zhao Fuqiang, Zhang Yanshun, Wu Minye, Zhang Yingliang, Xu Lan, and Yu Jingyi. Editable free-viewpoint video using a layered neural representation. In ACM SIGGRAPH, 2021.
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. arXiv preprint arXiv:2111.13539, 2021.
Alexander Kirillov, Yuxin Wu, Kaiming Hc, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 9799-9808 2020.
Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. arXiv preprint arXiv:2205. /5585, 2022.
Adarsh Kowdle, Dhruv Batra, Wen-Chao Chen, and Tsuhan Chen. imodel: interactive segmentation for object of interest 3d modeling. In European Conference on Computer Vision, pp. 211-224. Springer, 2010.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: seman- tic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12871-12881, 2022.
labelme. labelme.
Weihao Li, Omid Hosseini Jafari, and Carsten Rother. Deep object co-segmentation. Asian Conference Computer Vision, pp. 638-653. Springer, 2018.
David Lindell, Julien Martel, and Gordon Wetzstein. Autolnt: Automatic integration for fast neural volume rendering. https///xxi.oog///b2222220011114 2020.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Advances Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yat Zhu, and Bryan Russell. Editing conditional radiance fields, 2021.
Xinhang Liu, Jiaben Chen, Huai Yu, Yu- Yu-Wing Tai, and Chi-Keung Tang. Unsupervised multi-view object segmentation using radiance field propagation. arXiv preprint arXiv:2210.00450 2022.
Stephen Lombardi, Tomas Simon, Gabriel Schwartz. Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. 2021.
Ricardo -rrtin- Noha Radwan, Mehdi Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and
Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. https://arrivvorg/ab//00000.. 02268, 2020.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics (TVCG), 1995.
Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6351-6366 2021.
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions Graphics (TOG), 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421 Springer, 2020a.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020b.
11

---

Published as a conference paper at ICLR 2023
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. DeRF: Decomposed radiance fields. https:/aari.orrgbbb2111.1.4 2020.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In IEEE International Conference on Computer Vision (ICCV), 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 10901-10911, 2021.
Zhongzheng Ren, Aseem Agarwala, Bryan Russell, Alexander G Schwing, and Oliver Wang. Neural volumetric object selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6133-6142 2022.
Gernot Riegler and Vladlen Koltun. Free view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Carsten Rother, Tom Minka, Andrew Blake, and Vladimir Kolmogorov. Cosegmentation of image pairs by histogram matching-incorporatee a global constraint into mrfs. In 2006 IEEF Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'06). 066), volume 1, pp. 993- 1000. IEEE, 2006.
Pedro Savarese, Sunnie SY Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Information-theored segmentation by inpainting error maximization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4029-4039, 2021.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3D-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020a.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advunces in Neural Information Processing Systems, 33: 20154-20166, 2020b.
Tong Shen, Guosheng Lin, Lingqiao Liu, Chunhua Shen, and Ian Reid. Weakly supervised semantic segmentation based co-segmentation. In BMVC, 2017.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and 2021.
labels. arXiv preprint arXiv:2109.1420:
Cameron Smith, Hong-Xine Xing Yu, Sergey Zakharov, Fredo Durand, Joshua Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205. 03923, 2022.
Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01145: 2021.
Cheng Sun, Min Sun, and Hwamn- Chen. Direct voxel grid optimization: Super-fast fast conver- gence for radiance fields reconstruction. arXiv preprint 1rXiv:2/11.1122 2021.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419 Springer, 2020.
Alex Trevithick and Bo Yang. GRF: Learning a general radiance field for 3D scene representation and rendering. htmps://axiv.orgaaaz20000..44
2020.
Vadim Tschemezzi,, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. arXiv preprint arXiv:2209.03494 2022.
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. arXiv preprint arXiv:2201.00 004224 2022.
12

---

Published as a conference paper at ICLR 2023
Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, et al. Is attention all nerf needs? arXiv preprint arXiv:2207.13290. 2022.
Sara Vicente, Carsten Rother, and Vladimir Kolmogorov. Object cosegmentation. In CVPR 2011, pp. 2217-2224. IEEE, 2011.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprint arXiv:2111. 2021.
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. arXiv preprint arXiv:2112.05139, 2021a.
Qianqian Wang. Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Bauaala Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 134))6600-612 2004.
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXII, pp. 736-753. Springer, 2022.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13779-13788, 2021.
Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 879-888 -888, 2019.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1790-1799, 2020.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In IEEE International Conference Computer Vision (ICCV), 2021a.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Hong-Xing Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint 11X.9:200,.07905, 2021c.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492. 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 586-595, 2018.
13

---

Published as a conference paper at ICLR 2023
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. -6269-16268, 2021.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 55838-15844 2021.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.I ADDITIONAL TRAINING DETAILS
Implementation of the Patch Selection We reconstruct positive and negative pairs on the fly during training. Given N rendered patches from N different viewpoints in training, we fed the patches into the DINO- ViT and obtained the [CLS] tokens. Next, we compute a N x N similarity matrix using the cosine similarity with the [CLS] tokens. The negative pairs selected from the pair with the lowest similarity in each row; the positive pairs are set as the identity pairs. Overall, 2N pairs (N positives N negatives) are formulated per iteration compute the collaborative contrastive loss.
Implementation of Stride Ray Sampling Neural radiance field casts number of rays (typically not adjacent) from the camera origin, intersecting the pixel, to generate input 3D points in the viewing frustum. Our model requires patch-wise rendering of size (P, P) to formulate the collaborative contrastive loss. However, we can only render a patch less than 64 x 64 in each view due to GPU memory bottleneck (Garbin et al., (02T). Thus, it hardly covers sufficient receptive field to capture the global context, using the pre-trained DINO. To solve this problem, we adopt Strided Ray Sampling strategy (Schwarz et al., 2020b) Meng al. 202I), to enlarge the receptive field of the patches while keeping computational cost fixed. Specifically, instead of sampling a patch of adjacent locations (P x k) x (P k).
P, we sample rays with an interval k, resulting in a receptive field of
Hyperparameters Selection The hyperparameters of NeRF-SOS on different datasets are shown in Table 5 We adopt the number of b and Gaelf in (Hamilton et al. 2022 as the hyperparameter for our appearance level loss (bneg and btd, respectively.). We share the appearance level hyperparameters across all datasets. We set the weights of baeg and 6dd in the geometry level loss with physical intuition. For example, since the radius of the foreground object in LLFF datasets is roughly 0.5 meters, we set the brey and bid to be 0.5 and 3, respectively. Analogously, for the unbounded scene (e.g., scene Truck), the geeg and l are set to be and 5, respectively.
Table 5: Hyperparameters of NeRF-SOS different datasets. We share the hyperparameters of appearance level for all datasets while we set the gneg and 6dd for geometry level loss with physical intuition.
paraneter name
LLFF
BlendedMVS
CO3Dv2
Tank and Temples
bialC Cgenn
0.50
0.12
0.25
1.00
Breng Cgeo))
3.00
0.60
1.00
5,00
Self-supervised Learned 2D Representations We adopt DINO-ViT ViT (Caron et al. 2021 as our feature extractor for distillation. The training process of DINO- ViT largely simplifies self-supervised learning by applying a knowledge distillation paradigm (Hinton et al.] 2015) with momentum encoder (He et al. 2020). where the model is simply updated by a cross-entropy loss.
A.2 HUMAN ANNOTATION DETAILS
We use the publicly available annotation tool: (labelme) for the foreground and background annotation. To be specific, we annotate all training and testing views of different scenes using ponygo,,, extract the polygons and convert them to binary masks. The masks of scene Flower are included in our supplementary and we provide usage guidelines in README. All annotated data will be made public.
A.3 ADDITIONAL EXPERIMENTS
Comparisons with Semantic-Nerl using Sparse Label As Semantic-NeRF ables to perform label propagation with sparse annotation, simulate sparse user annotation by randomly applying {1, 1 1%, 5%, 10%) foreground annotated object pixels while leaving the rest unlabeled. We can see from Figure 9 the foreground boundaries gradually refined when more annotations included, which
15

---

Published as a conference paper at ICLR 2023
Table 6: Comparisons with Semantic-NeRF using sparse labels. Results are calculated scene Fortress
Scene "Fortress"
User Click
NV-ARI f loU(BG) t
loU(FG) t
mloU t
Semantic NeRF
0,9615
0,9812
0,9528
0.967
Semantic NeRF
1%
0,9731
0.9938
0.9667
0.9803
Semantic NeRF
5%
0.9783
0.9950
0.9731
0.9841
Semantic NeRF
10%
0.9788
0.9952
0.9735
0.9843
Semantic NeRF
100%
0.9838
0.9963
0.9799
0.9881
NeRF-SOS
0
0.9802
0.9955
0.9751
0.9853
Mask Annotations
NeRF-SOS
Semantic-NeRF (1 click)
Semantio-NeRF
Semantic-NeRE
Semantic-NoRF (10% click)
Semantic-NeRF (100% click)
(1% click)
(8% click)
Figure 9: Visual comparisons among NeRF-SOS and Semantic-NeRF with sparse annotations.
Table Comparisons among several NeRF-based methods for object segmentation.
LLFF Dataset
Supervision Type
NV-ARI f
IoU(BG) t IoU(FG)
mloU
RFP
Self-supervised
0.9267
0.9812
0,9178
0.9495
NeRF-SOS NVOS ObjectNeRF
Self-supervised
0.9665
0.9912
0.9627
0.9769
Weak-supervised
0.9217
0.9793
0.9145
0.9469
Supervised 0.9666
0.9909
0.9639
0.9774
consistent with reported results in the original paper of Semantic-NeRF. However, we can see from Table all sparse annotation experiments show insufficient accurate salient foreground segmentation, compared with dense annotated Semantic-NeRF: Whereas NeRF-SOS performs comparably with the dense annotation counterpart.
Comparisons with other NeRF-based Object Segmentation Label-free NeRF segmentation on real-world scenes remains a challenging problem. ObjectNeRF (Yang et al. (022I) appends an object branch input with object feature and object code, with object-level supervision (a.k.a. 2D instance masks) to enable object manipulation after training. Although Neural Volumetric Object Selection (NVOS) (Ren et al.) 2022) solves interactive ceakkssuprriiedd object segmentation, it requires users to provide several scribbles for supervision. The concurrent work, RFP (Liu et al. 2022), tackles the label-free NeRF segmentation on real-world scenes, but there is still room for their segmentation accuracy. To handle the high-quality label-free NeRF segmentation on real- world scenes, NeRF. SOS leverages the proposed collaborative contrastive loss to do self-supervised object segmentation. Experiments the following Tables and Figure 10 demonstrate our label-free method consistently outperforms RFP and weakly-supervised NVOS.
Qualitative Visualization on More Views Qualitative comparisons on LLFF dataset can be found Figure IT and Figure 12 respectively. Qualitative comparisons on BlendedMVV dataset can be found in Figure 13 and Figure 14 respectively. Qualitative comparisons CO3Dv2 dataset can be found in Figure 15 and Figure 16, respectively. Qualitative comparisons on Tank and Temple dataset can be found in Figure Here, visualize three different views to show the segmentation consistency across views. Visualized video can found in the supplementary.
16

---

Published as a conference paper at ICLR 2023
Color Image
Mask Annotations
NeRF-SOS
RFP
NVOS
ObjectNeRF
Figure 10: Visual comparisons of the segmentation masks among NeRF-SOS (self-supervised), RFP (self-supervised), NVÓS (wcakly-supervised), and ObjectNeRF (fully-supervised),
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 11: Novel view object segmentation results on scene Flower of LLFF dataset.
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 12: Novel view object segmentation results on scene Foretress of LLFF dataset.
17

---

Published as a conference paper at ICLR 2023
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 13: Novel view object segmentation results on scene 5a3 of BlendedMVS dataset.
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 14: Novel view object segmentation results on scene 5a6 of BlendedMVS dataset.
18

---

Published as a conference paper at ICLR 2023
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 15: Novel view object segmentation results on scene Apple of CO3Dv2 dataset.
19

---

Published as a conference paper at ICLR 2023
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 16: Novel view object segmentation results scene Backpack of CO3Dv2 dataset.
Color Images Mask Annotations
Ours
DINO-CoSeg
DOCS
Sem.NeRF
Figure 17: Novel view object segmentation results on scene Truck of Tank and Temples dataset.
20


Reviews:
[{"review_id": "cgSN0TL9LTZ", "reviewer": "Reviewer_LCEX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an appearance contrastive loss to apply the self-supervised learned 2D visual feature for 3D representations.\nA new geometry contrastive loss is proposed for object segmentation to involve geometric information for segmentation clustering.\nThe segmentation of the proposed method has a more refined result than its supervised counterpart.", "summary_of_the_review": "The paper is well-written, and the result of object segmentation is better than its counterpart obviously.\nI think it is marginally above the acceptance threshold.", "strength_and_weaknesses": "Strengths:\n1. This is a well-written paper.\n2. The proposed method is compared with various methods. The experiments are complete and convincing.\n3. Some visualizations are helpful to understand.\nWeaknesses:\n1. Although the visual results of the proposed method are better than the supervised method. The evaluation metrics of the proposed method are worse than Semantic-NeRF in Tab 2 and Tab 3. What causes that problem?\n2. The paper lacks a quantitative analysis of the effects. May you give an analysis of your design?\n3. The ablations are not sufficient enough. Ablation should contain all of your extra design including geometry contrastive loss and appearance contrastive loss.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SZeABaIXEYM", "reviewer": "Reviewer_vb78", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a self-supervised object segmentation technique that operates on an optimized Neural Radiance Field. It couples NeRF reconstruction with DiNO-ViT feature extraction and models a collaborative contrastive loss function that helps parse the dominant object in the image. The results show that the segmentation accuracy is at par with the semi-supervised Semantic NeRF and outperforms other approaches. ", "summary_of_the_review": "The paper proposes a novel idea of object segmentation by coupling the NeRF color and depth features with DINO powered semantic segmentation features. It provides a nice demonstration that the dominant object in the NeRF model can be parsed in a full self-supervised manner. The details of how to go about it are a bit sketchy, and the lack of detailed ablation studies makes it harder to understand the merits and the inner workings of the correlation volumes. Adding more diverse results of complex real-world scenes, and teasing apart the results from the DINO model alone would make the results more interesting. \n\n", "strength_and_weaknesses": "Since NeRF is trained under an i.i.d. assumption, there is little semantic correlation between the features across different views. Hence, the color or depth MLPs can't be directly used for semantic tasks. In this paper, a new semantic branch is added to the network that is trained by explicitly enforcing matches across views through collaborative contrastive losses. This is a novel idea, and it becomes feasible to extract the dominant object simply by clustering the logits of the semantic branch. This appears to be a really effective approach and is able to match the segmentation quality of semi-supervised methods such as the Semantic-NeRF. \n\nAlthough the method claims to segment complex real-world scenes, the demonstrated examples are typically single object scenarios with no background noise. In most cases the objects are also centered and parallel to the visual plane. The authors mention that a density based correlation is required to distinguish between objects, but neither the qualitative nor the ablation results reflect the need for this additional loss. The results could be much more interesting if these design choices and other parameters such as the number of kmeans clusters were explained. \n\nWhen a high-performance model such as DINO-ViT is used, it becomes difficult to tease apart the value of the core method vis-a-vis the strong model. The results of the proposed method with Resnet seem much less impressive that with DINO backbone. How much does the DINO model alone contribute to accurate segmentation?\n\nThe comparison to Semantic NeRF is also a bit misleading, since this model can segment only a single dominant object that's in the field of view, whereas Semantic NERF allows simultaneous segmentation of many class labels.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SB7cK8zZpFA", "reviewer": "Reviewer_92aZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a self supervised method for object segmentation in NeRF. The major idea is applying collaborative contrastive training in both appearance (radiance field) and geometry (density field) level in NeRF. It uses DINO (Amir et al., 2021)  as the basic backbone architecture to extract appearance correlation volume and extend it with geometry correspondence across views. The proposed method is evaluated on several NeRF dataset and show that it outperforms other cosegmentation methods, i.e. DINO-CoSeg and DOCS, that were designed for multiple image cosegmentation but not for NeRF setting.", "summary_of_the_review": "Please check my above comments regarding missing references and comparisons, and the incremental novelty of the proposed method.", "strength_and_weaknesses": "Strength: It is a good attempt to extend previous cosegmentation method for NeRF setting. The proposed method is reasonable and the results are better than the compared methods.\n\nWeaknesses: The proposed method is not the first work that deal with segmentation in NeRF. Some important references are missing. The experimental comparisons are incomplete as it miss the comparisons with some recent methods. Please check the following recent works:\n\nYu et al., Unsupervised Discovery of Object Radiance Fields, in ICLR, 2021 (missing comparisons)\n\nYang et al., Learning object-compositional neural radiance field for editable scene rendering, in ICCV 2021  (missing comparisons)\n\nRen et al., Neural Volumetric Object Selection, in CVPR 2022 (missing reference and comparison)\n\nSmith et al., Unsupervised discovery and composition of object light fields, in ICLR, 2022. (missing reference and comparison)\n\nKundu et al., Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation, in CVPR, 2022. (missing reference)\n\nLiu et al., Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation, in NeurIPS 2022. (This one should be considered as a concurrent work since NeurIPS conference is after ICLR deadline.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AIhdoVC-XUd", "reviewer": "Reviewer_9Jcm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a self-supervised approach to segment objects from novel views synthesized from a NeRF.\n\nThe idea is borrowed from Hamilton et al. (2022) who learned semantic co-segmentation of images in an unsupervised fashion, by distilling the knowledge of DINO. The authors adapted the method to NeRF, to segment objects in novel views without any extra supervision. The knowledge is distilled from both visual and geometric features learned by NeRF.\n\nThe novel object segmentation branch in NeRF, learned by self-supervised distillation of the visual and geometric knowledge, show competitive performances with fully supervised methods such a Semantic NeRF, that require extra manual annotation of the masks.\n", "summary_of_the_review": "The method provides very interesting insights on how to learn object segmentation in an unsupervised approach, leveraging consistency between multiple views in a NeRF representation.\n\nThe work appears technically sound, and paves the way for further work towards less supervised learning segmentation.\n", "strength_and_weaknesses": "Strengths:\n\nThe paper is well written and the method is technically sound. \n\nThe authors accommodated for a segmentation branch in the NeRF architecture, and the learning for that branch does not require any supervised training.\n\nThe results appear on par with fully-supervised approaches like Semantic-NeRF.\n\nWeaknesses:\n\nIn the second phase of the training, it is unclear whether the color and density heads of the NeRF are frozen or not. Even though the \\lambda_0 is set to 0, the other 2 losses can still induce a catastrophic forgetting of the knowledge learned in the 1st phase.If those are not frozen, the 2nd stage of the training might lead the appearance and geometry head of NeRF to collapse instead of distilling their knowledge into the segmentation head.\n\nThe cross-view geometric correspondence is unclear, in particular the motivation and insight on Equation (7), that appears to average the geometric coordinate t_k along the ray, based on the volumetric rendering based on the density learned by NeRF. Is that simply the point on the surface? Why not integrating some higher dimension features learned in NeRF? Also, for correctness, I believe \\Delta_k in (7) should be \\delta_k like in (1). Moreover, Equation (8) introduces spatial features g and g’ of dimension c, why is that the same dimension of f and f’ introduced in (3) and not of 3 dimension? Is the absolute distance an L2 norm?\n\nIn Equation (5) and (10), why \\lambda_neg is not a negative value?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster