Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation
Authors: Xinhang Liu
Jiaben Chen
Huai Yu
Yu-Wing Tai
Chi-Keung Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
Unsupervised Segmentation
3D Scene Representation
Computational Imaging

Abstract: We present radiance field propagation (RFP), a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene. RFP is derived from emerging neural radiance field-based techniques, which jointly encodes semantics with appearance and geometry. The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient or meaningful regions corresponding to different object instances. To better handle complex scenes with multiple objects and occlusions, we further propose an iterative expectation-maximization algorithm to refine object masks. To the best of our knowledge, RFP is the first unsupervised approach for tackling 3D scene object segmentation for neural radiance field (NeRF) without any supervision, annotations, or other cues such as 3D bounding boxes and prior knowledge of object class. Experiments demonstrate that RFP achieves feasible segmentation results that are more accurate than previous unsupervised image/scene segmentation approaches, and are comparable to existing supervised NeRF-based methods. The segmented object representations enable individual 3D object editing operations. Codes and datasets will be made publicly available.

Unsupervised Multi-Vin View Object Segmentation Using Radiance Field Propagation
Xinhang Liu' HKUST
Jiaben Chen UC San Diego
Huai Yu
Yu-Wing Tai
Chi-Keung Tang'
Wuhan University
KKuaihhou Technology
https://xinhangll wwt.:/cfmooeeeeeeeeee
Abstract
We present radiance field propagation (RFP), novel approach segmenting objects in 3D during reconstruction given only unlabeled multi-view images of scene. RFP is derived from emerging neural radiance field-based techniques, which jointly encodes semantics with appearance and geometry. The core of our method a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient meaningful regions corresponding to different object instances. To better handle complex scenes with multiple objects and occlusions, we further propose an iterative To the best of our knowledge, RFP is one of the first unsupervised approaches for tackling 3D real scene object segmentation for neural radiance field (NeRF) without any supervision, annotations, or other cues such 3D bounding boxes and prior knowledge of object class. Experiments demonstrate that RFP achieves feasible segmentation results that are more accurate than previous unsupervised image/scene segmentation approaches, and are comparable to existing supervised NeRF-based methods. The segmented object representations enable individual 3D object editing operations.
-anittiiiimaiiiiiitiiio algorithm to refine object masks.
Introduction
Despite deep learning's remarkable success in object detection and segmentation 33 31 over the past decade, several issues remain relevant. First, modern neural network approaches require a massive amount of labeled, heavily curated data [ [9 to achieve superior performance, or they may have difficulty dealing with problems in which labeled data are lacking. In addition, most segmentation methods regard images as flat 2D arrays of pixels where pixel groupings form object instances. Natural images 2D projections of the underlying 3D world. Thus, discarding of the dimensions gives rise to fundamental ambiguities and challenges to the task resulting from partial or total occlusion, especially for single-image image methods.
The above fundamental issues motivate us to reconsider the problem of object detection and segmen- tation from 3D perspective. In this paper, we study the problem of automatically partitioning a 3D scene into an arbitrary number of salient or meaningful regions, given only 2D images of the scene from multiple viewpoints.
Estimating semantic labels 3D scene closely related predicting geometry and appearance. Therefore, recent success in novel view synthesis achieved by Neural Radiance Field (NeRF) 23 and many follow-up work 224 46 9 will likely make long-lasting impact for semantic scene understanding. For example, SemanticNeRF (51) extends radiance fields to jointly encode semantics with appearance and geometry; its multi-task learning setting enables smooth and coherent semantics prediction. Nonetheless, supervised approach, SemanticNeRF requires semantic labels for full
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Propagated RF of objert
Correctly rendered image
Ray Disunct
Ground truth image
Propugaiod RF for object
Bay Disases
Frroneously rendered image
Density
Rendering with -..
Minimize Error
3D mask m,
Rendering with co
Maximize Frror
semantic field
(b)
Figure 1: Overview of radiance field propagation (RFP). (a) Semantic field and propagated radiance field for object O (background) and object (foreground); (b) volume rendering with correct and erroneous coloring. followed by applying the bidirectional photometric loss. The 3D semantic field is learnt in such an unsupervised manner. For clarity, we show here a single salient foreground object example.
supervision. When labels incomplete noisy, this approach still relies cross-entropy loss between the prediction and the target label to supervise its semantic branch. Previous work [48] and concurrent work (36] studied unsupervised discovery of object field to enable 3D scene segmentation and editing. But their system needs pre-training on datasets with specific object categories, such as CLEVR (14) and ShapeNet [4] objects, thus capable of segmenting arbitrary objects in general scenes.
To explore fully self-supervised image segmentation in the multi-view setting, study the task of partitioning a scene into multiple object instances with only multi-view images and calibrated camera poses. In this setting, annotated labels in any form not available. Furthermore, there are other cues, such as 3D bounding boxes for the objects or prior knowledge about object classes.
To tackle this challenging task, we propose radiance field propagation (RFP) based on the emerging radiance field-bace based scene representation. Fig. [ shows an overview of our method. We use continuous 3D semantic field like [51] to model semantic information in space. Note that we do not model object geometry and appearance by a single unified radiance field, rather by a layered scene representation similarly adopted in [49] and [43] representing each entity (each foreground object and the background) as an independent radiance field. Based on layered scene representation, we introduce a propagation strategy each radiance field. The input to these continuous radiance field functions can be (a) a coordinate occupied by the corresponding object, or (b) a coordinate occupied by other objects or object. For (a), we expect the function's 's output to be accurate model of the object's ' geometry and appearance. We propagate the statistical bias of appearance from (a) to (b). (Note that the density function or geometry is not propagated.)
Inspired from the unsupervised object detection and segmentation approaches 45 that accurate partitioning of the scene should make the estimation of one part from others difficult. we use the semantic field to distinguish (a) and (b), then an accurate semantic field should maximize the error between the propagated appearance from (a) to (b) and the ground truth appearance. A bidirectional photometric loss term thus proposed that maximizes the error of crroncously rendered image from propagated appearance, while minimizing the error of correctly rendered image, that self-supervision of the learning of the semantic fields can achieved.
we assume
While the above techniques can segment well scenes with single salient object, to handle more complex scenes with multiple objects and occlusions, we design post-procesn post-procsssing procedure to refine object masks based on iterative expectation maximization. sxpetation-maxxiizztonnn These procedures improve the 3D segmentation of each object scenes with multiple objects. With individually segmented object NeRFs, we can render individual objects and even enable various 3D scene editing, including object insertion, duplication, translation, and rotation, which is achieved by the previous works when using manual annotation.
49 43 only

---

As the first pertinent unsupervised multi- image method, the experiments show our system performs better than the SoTA methods for unsupervised single image segmentation or 3D scene segmentation, and is comparable with supervised NeRF-based novel view semantic synthesis approaches. In summary, the contributions of this paper are follows.
We are among the first to study fully unsupervised multi-view image segmentation with real scene NeRF without any annotations such as 3D bounding boxes, or prior knowledge of object class to the best of our knowledge.
We exploit layered radiance fields and propose Radiance Field Propagation with bidirec- tional photometric loss to guide the training the semantic field.
We design an iterative handling complex scenes.
cataattioonmaiiizatiiin algorithm to refine object masks for
Extensive experimental validation and extensive ablation studies justify the design of each component and demonstrate the effectiveness of our system on applications such as individ- ual object rendering and editing.
2 Related Work
Unsupervised Single Image Segmentation. Despite the success of supervised deep learning based methods [20] 32 [ (50 [[5). much effort has been made on unsupervised single image segmentation. Different from supervised image segmentation, where pixel-level semantic labels such floor or table, unsupervised image segmentation aims to predict more general labels, such as foreground and background. To solve unsupervised category-agnostic image segmentation, which could be regarded as primarily a grouping or clustering problem, as opposed to a labeling task, using color, contrast, and hand-crafted features cluster pixels has been investigated [7 15 [1].
More recent approaches
use generative models to partition an image 66 2].
Notably, in Contextual Information Separation (CIS) [45] object detection and segmentation from an information-thhoretiic perspective were first studied. To detect moving objects in images, CIS uses segmentation network to minimize the mutual information between the inside and the outside of putative motion regions. This procedure is however sensitive to motion errors. To further enable stationary objects detection and segmentation, DyStaB 44 partitions the motion field by minimizing the mutual information between segments which are then used to learn object models. While in inpainting is discussed any function of the image (color histogram or optical flow), later in 33 (with somewhat similar technicalities to (45)) the authors consider inpainting on just the image itself and does not involve training deep network with external data.
CIS 45
Nevertheless, all these works perform segmentation on a single image basis without considering the 3D information of the scene underlying the 2D image. In addition, the class-agnostic segmentation
approaches can only tackle images with single salient object such as those from (39 47 26 we consider scenes with an arbitrary number of salient objects.
while
Radiance field-based scene representations. Our unsupervised segmentation work builds on Neural Radiance Fields (NeRF) 23 which represents a scene using a multi-layer perceptron (MLP) that maps positions and directions to densities and radiances. Following work [46] 9 3] improve NeRF for faster training and inference and more realistic rendering. Dynamic neural representations enable 4D reconstruction and rendering 29 49 28 40). Using MIP (23) or explicit feature grids 9 R) these radiance field-based scene representations achieve unprecedented novel view synthesis effects. Our system can be built upon any radiance field-based scene representation.
NeRF with semantics and object decompositions. Previous works have investigated inferring semantics and achieving object-level scene understanding using NeRF [5T]. In particular, Semantic- NeRF [5T] adds an extra head to NeRF to predict semantic labels at any 3D position. Recent work PNF 18 can estimate a panoptic radiance field representation of any scene from just color images. However, SemanticNeRF [5]] takes 2D semantic label maps as input, while PNF [88] is brser- and trained with external annotated data.
10 a scene was decomposed into a set of NeRFs associated with foreground objects separated from the background. However, [49 43
449
cannot produce 3D segmentation of objects to enable rendering and editing of individual objects. ST-NeRF(49 and ObjectNeRF [3 represent each entity with separated NeRFs. While enabling
3

---

rendering and editing of a single object, they require semantic information provided by the label map for supervision. GIRAFFE [25] represents scenes as compositional generative neural feature fields, enabling disentanglement of individual objects for image synthesis without studying obtaining 3D scene segmentation. [37 turns single image of a scene into 3D model represented as a set of NeRFs, with each of them corresponding to different object. They enable unsupervised 3D) scene segmentation and novel view synthesis, with experiments are on synthetic datasets. After that [48] and concurrent work [36] studied unsupervised discovery of object fields which enables scene segmentation and editing in 3D. But their systems are prior-bauc based and need pre-training on datasets with specific object categories, such CLEVR [14] and ShapeNet [41] objects, thus not capable of segmenting arbitrary objects in general scenes. Unlike all these methods, our method can segment objects in 3D scenes and perform individual object rendering and editing without any
semantic annotation and knowledge of object class.
3 Method
With a set posed images 3D scene as input, our goal is to reconstruct the scene and segment each of the object instances. We view each salient object in the foreground as an object instance, and the background an extra separate instance. We assume there are in total K salient foreground object instances in the scene (we denote foreground objects instance 1 to K and use instance 0 to denote the background as instance in addition).
3.1 Propagation of Layered Radiance Field
Extented from NeRF [23], represent the continuous volume density, emitted color, and seman-
tics [5]] as functions with a spatial coordinate x = (z, y, 2) and viewing direction d = (0, o) as input. Specifically, represent volume density as a function that maps world coordinate (x, v, 2) to the continuous 3D scene density a: (x) =
we formalize semantic information as an inherently hie-iinvariant function that maps only world coordinate x to a distribution over K + 1 labels corresponding to K + object instances via pre-softmax semantic logits s(x) (8o, .8):
Following [11,
s(x) F3 (x; θ).
Here only consider K labels for all practical instances, without an additional label for non- object coordinates of which the contribution would be eliminated automatically by a value close to 0 of FL (x). With semantie logits of each coordinate, we compute the label to assign coordinate by argmax over all the objects.
1(x)
Sargmax (x).
Since the operation of argmax is not differentiable, we use the straight through trick and compute the assignment as
12
i(x) one- (Sargmatr + s(x) sg(s(x)),
where sg is the stop gradient operator. i(x) has the one-hot value of assignment to object, whose gradient is equal to the gradient of s(x), thus making the system differentiable and allowing gradient back propagation. We obtain the 3D mask for object
{
1.
((x)
mx(x)) = 0, # k.
(5)
We further define गा 1 the inversion operation masks. Instead of unified function to represent the continuous field the emitted color the whole scene, we use K + 1 functions to model the appearance of K instances. Such individual modeling of each object with different function is similar to some previous work [49] (43] For object k, we represent its color as a function that maps world coordinate x (x, 2) as well view direction d (0, Φ) to the continuous 3D scene color = (r, g, b):
cx(x, F (x, d: +).
(6)

---

(b)
(d)
Figure 2: For clarity this example shows one salient foreground object. Object and respectively denote the background and the foreground object: (a) corrected rendered image, (b) image rendered with object I's radiance field, (c) image rendered with object 0's radiance field and (d) image with obiect O rendered with obiect I's RF and obiect 1 with obiect 0's RF.
(b)
(d)
Figure 3: Illustration Radiance Field Propagation on a multi-object scene. (a) corrected rendered image, (b) image rendered with object I's radiance field, (c) image rendered with object 2's radiance field and (d) image rendered with object 3's (background) radiance field
Starting here we use C₂ (x) instead of (x, d) for notation simplicity. The input world coordinate x this function can an arbitrary coordinate the scene, i.e., either occupied by object k or otherwise. The 3D object mask in Eqn. 5 naturally distinguishes these two cases. For the former case, mx(x) (x) = 1, and we expect cx(x) to be the precise color the foreground object to achieve realistic rendering results. In the latter case, m& (x)
O, and encourage (x) here to be propagated from coordinates in the former case and have value of color close to object k. This 3D propagation procedure is enforced by the propagation regularizer
Cprop
my (x](x((( 1)1l2
where y is the average value of my (x) (not counting masked out values). The average operation performed on the training batch. At coordinates occupied by object k. the propagated value is similar to one at a coordinate that belongs to object k which can be regarded as a prediction of the radiance of partition other than object k knowing object k's radiance field.
3.2 Bidirectional Photometric Loss
Since an accurate segmentation of object should maximize the error of this appearance prediction, a loss term Eqn. 12 encouraging such maximization of error would guide the optimization of the semantic field without supervision from any ground-truth annotations.
For each coordinate, we calculate the value of all K + 1 color fields at it. The correct assignment for this coordinate color is given by
c(x) = Σ my (x), k=0
(8)
If we artificially color object i with c, (x), and color objects other than object i with the color of object cx),,, we get deliberately erroneous color field
Ci (x) = mx)c (x) + (x),,((xx).
(9)
To compute the color of a single pixel, with either the correct or an erroneous radiance field, we approximate volume rendering by numerical quadrature, the same way [23]. Let r(t) + td
5

---

be the ray emitted from the centre of projection of camera space through given pixel, traversing between near and far bounds (t,, and ty), then for selected random quadrature points {te)p=) between in and f,, the approximated expected color using c(x) and the erroneous color using C2 are given by:
P E
C(r) T(t)) (o(t))))))))
(10)
p=1
(r)(t) Σ (o())),0
(11)
p=1
where T(t,) = exp (- (- - ot (t,)), (z) = exp(-2), and 6, = 1p+1 - tp is the distance between two adjacent quadrature sample points. We show examples of images using different rendering scheme in Fig. 2 and Fig. 3 To train our system, we minimize the error of the correctly rendered colors (Fig. ). and at the same time maximize the error of the erroneously rendered colors (Fig. d)). This yield to the bidirectional photometric loss function to train our system.
K
phooo = Σ (|0(r) (()t)))(r)(( FER
Σ
(t)
(12)
K(K +
KKMirErrr FER
where R are the sampled rays within a training batch. We compute the expected semantic logits of single pixel in an image the same way [51]:
S(r) = Σ (()))(((()))))())))
(13)
which can then be transformed into multi-class probabilities through softmax normalization layer. The 2D label map and object mask for object i are
L(r)= Sargax (r),
(14)
1.
L(r) k,
(x(r)
(15)
0,
L(r)
We also found that estimating a coarse 2D label map for each input image using an unsupervised single image-based approach and then computing cross-entropy loss between the map and the estimated semantic logits can improve the results of our system:
Cinit
Σ PER
Ske (r) log S* (r). min
(16)
where Sintt is the initial coarse label map. Since the approach use is unsupervised and does not involve external data, so this initial stage does not undermine this property of our method. Hence the total training loss L combining the bidirectional photometric loss Eqn. [2] the propagation regularizer
Eqn. 7 and the initial semantic estimation loss Eqn. 16 is:
L - Cphoto + Xpop + Ainit Linit
(17)
where Aprpp and Anii weights of corresponding loss terms.
3.3 EM-based Mask Refinement for Complex Scenes
In the previous subsection we have described the core components necessary for performing 3D seg- mentation, and our experiments have shown the effectiveness of the proposed system segmenting scenes with one salient object. To extend the segmentation on more complex scenes with multiple foreground objects and occlusion among them, further propose EM algorithm- based procedure to refine the object mask as processii.....
For each input image, we first feed it into deep feature extractor, and cluster all the pixels into two classes using k-means clustering. this way we get initial 2D image masks for fore- ground (consisting of K objects) Storoground and the background Spackground- We apply differentiable

---

feature clustering (DFC) [17] to further partition foreground masked by Storeground into K parts, { int J.m......KK and let S backgrond: for computing the initial semantic estimation loss Eqn. 16 Note that here we introduce deep feature extractor [S] trained large-scale data, which is the only introduction of external data to our method. But since no annotated data is involved, the unsupervised nature of our method is not undermined. DFC results in response vector map (ar) for each image, where the is a pixel index. Pixels belonging to the same object tend to have similar response vectors. After the training of our system, anpectationmmaximizaiin (EM) is employed to refine the output mask obtained from Eqn.
15 with these response vector maps. Let 24g be a mean
vector of object and we initialize 2 as
E. Ma(x)v(a)
Mi
(0)
En Ma(a)
(18)
We formulate the posterior probability of v given M4 and the latent variable εy at 1-th EM iteration as
= expl-1lv - 1xll)),
(19)
p(v(x))m4 (
(x) P(o
(20)
At each EM iteration, we update the μ as
E, (4) (x)v(a)
(1+1)
Pk
(21)
Σ *k (x)
The final semantic logits are given a weighted sum of Z concatenated from {=x( K at T'-th iteration and estimated logits after training in Eqn. 13
Smual Z + wS.
(22)
where w is a weight. Therefore, the refined mask can be computed similarly to Equ. 14 and Equ.
With scene segmentation obtained using our system, applications such as single object rendering and editing can be achieved. We defer the description of the relevant procedure in the supplemental material.
4 Experiments
We evaluate method on three datasets with different complexities. We report qualitative and quantitative results on scene segmentation with comparison to previous work. We also show the application of our method on individual object rendering and editing. We conduct ablation studies to justify each component of our method. We gently urge readers to check the supplementary material for more qualitative results in the form of pictures and videos, well the settings of our experiments in detail.
Datasets. We first test RFP scenes with single foreground object using two real datasets, Local Light Field Fusion (LLFF) [22] and Common Objects 3D (CO3D) 30 As LLFF does not provide ground-truth label maps, we only show qualitative results on the LLFF dataset. To test RFP on scenes with multiple objects, we build a synthetic dataset upon ClevrTex [16]. We split all the datasets into training views and testing views with ratio of around 9 to 1.
Metrics. We adopt the widely-used pixel classification accuracy (Acc.) and mean intersection over union (mloU) as our metric. To evaluate scene segmentation in 3D, we report the metrics computed for both training and novel views. The former enables direct comparison to 2D methods, and the latter reflects the 3D nature. We denote Acc. and mloU of novel views as N-Acc, and N-mloU. Note that for our method, the difference between the training and the novel view only related to whether the color image of the view is provided. The semantic label of any view is not input. Differently, for the supervised methods (51 43), both the color image of the training view and the ground truth semantic label are provided as inputs.
7

---

NIA
N/A
NIA
N/A
Ground truth color image
Ground truth segmentation
SemanticNeRF
ReDO
SIRF (ours)
Figure 4: Qualitative comparison on 3D segmentation on scenes with a single foreground object.
IEM [33] and ReDO [6] are unsupervised single image-based methods. There are ground truth
labels for the L.LFF dataset [22], and thus the supervised approach SemanticNeRF (1 is not
applicable. Images are cropped for typography.
CO3D 30
RFP Synthetic
Methods
Acc.
mloU 1
N-Ace, f
N-mloU1
Acc.
mloU f N-Acc.
N-mloU
ReDO 6
86.7
63.9
IEM
33
95.8
83.4
DFC
17
95.7
79.4
oORF [48
91.7
32.6
92.1
27.6
SemantieNcRF 51]
99.0
96.0
98.7
95.1
99.6
96.6
99.3
94.8
ObjectNeRF 43
99.6
96.6
99.2
94.5
RFP (W/O PROPAGATION)
96.2
84.5
95.4
57.7
84.6
85.3
RFP
76.3
49.4
73.5
49.0
60.8
94.1
55.3
RFP (w/o EM REFINEMENT)
96.8
86.5
97.1
86.9
RFP (nurs)
98.5
94.1
97.8
94.0
97.7
88.0
97.4
87.5
Table I: Quantitative comparison and ablation. N-Acc. and N-mloU are Acc. and mloU evaluated on novel views. The best results without supervision are boldfaced.
4.1 3D Segmentation of Scenes with Single Foreground Object
Because there no previous work with the same unsupervised multi-viww setting as our work, we compare with SoTA unsupervised class-agnostic segmentation methods, ReDO [6] and IEM [33], which ensembles grouping or clustering problem with single images without considering 3D nature. We also compare RFP with supervised novel view semantic synthesis method SemanticNeRF [5].
We show quantitative results in Tab. [ and qualitative results in Fig. outperforms all single 2D image-basee methods on all metrics. Qualitatively, compared to single 2D image-based methods 33 [), our method's results have sharper and smoother edges, with no holes in the interior of objects due to the 3D nature of the scene. Our results are even comparable to the supervised method SemanticNeRF
Quantitatively, RFP
on the CO3D dataset 30 which may suffer from the noisy ground truth masks of the CO3D.

---

Inpur view
Novel view
N/A
Input view
Novel view
N/A
Ground truth color image
Ground truth segmentation
ObjectNeRR
DFC
uORF
SIRF (ours)
Figure 5: Qualitative comparison multi-object scene segmentation in 3D RFP synthetic dataset. Novel view color images are for reference and are not input. Images are cropped to typography.
4.2 3D Segmentation of Scenes with Multiple Objects
When tackling scenes with multiple objects and occlusion, we further perform the EM-based refine- ment introduced in Sec. 3.3 While using deep feature extractor trained on large-scale data means introducing external data to our system here for the only time, since no annotated data is involved, the unsupervised nature of our method is not undermined.
We compare RFP with the unsupervised segmentation method DFC [77] Since DFC relies on an initial separation of the foreground and the background, we perform k-means clustering before- hand. Such k-means clustering followed by DFC segmentation also the pre-processing procedure mentioned in Sec. 3.3 We also compare RFP with unsupervised object radiance field discovery (uORF) as well as supervised NeRF-hased methods: ObjectNeRF (43] and SemanticNeRF [51].
We show quantitative results in Tab. I and qualitative results in Fig. 5 Quantitatively, RFP outperforms all unsupervised image/scene segmentation methods all metrics. Qualitatively, single 2D image-based methods [17 miss more valid detection than ours. uORF can hardly give good results, probably because the method was pre-trained on a dataset consisting of naive scenes and thus with large domain gap with our evaluation data.
4.3 Individual Object Rendering and Scene Editing
Following the procedure detailed the supplemental material, our approach achieves depth- aware and photo-realistic free-viewpoint single object rendering and editing results. Fig. 6 and supplemental matierials show examples of such applications. Fig. (a) (d) are color images
rendered from novel views. Fig. T (b) & (e) show single object rendering. Fig. (c) shows the editing
effect of duplicating and repositioning the duplicated blue chairs. Fig. (f) shows the editing effect of rotating the helicopter.

---

(c)
(d)
Figure 6: Examples of individual object rendering and editing as an application of our approach. (a) & (d) rendered novel view of the scenes. (b) & (e) single object rendering. (c) & (f) object editing operations such as duplication, translation and rotation.
4.4 Ablation Study
Here, we evaluate the performance of different components and loss terms in our approach. Let w/o PROPAGATION, w/o INIT. EST. and w/o EM REFINEMENT denote the respective variations of our approach without the radiance field propagation and using vanilla photometric loss instead of the bidirectional loss, without the initial semantic estimation loss, and without the EM refinement. As shown in Tab. quantitativel and in the supplementary material qualitatively, our complete approach achieves more accurate results than the above variations. The radiance field propagation and the bidirectional photometric loss ensure good utilization of the 3D nature of scenes, which helps to avoid ragged edges, holes in the object's interior, and false-posine possiivve estimates outside of the object. Although unsupervised single image segmentation approaches are shown to be not robust in most of the cases, it is still essential for our system as valid initialization. The M-sddaeed refinement further brings good edges when tackling complex scenes with multiple objects.
5 Limitation
As the first approach dealing with unsupervised multi-view segmentation, our method may have several limitations. To begin with, our method may have difficulty when the object of interest is severely occluded, textured or with similar appearance to the background. mainly takes advantage of appearance instead geometric cues. Thus using generative method to perform geometry inpainting may lead to a fruitful direction for future exploration. Third, the dependence pretrained feature extractor should be alleviated.
addition, our work
6 Conclusion
In this work, we present one of the first approaches in real scene NeRF for tackling unsupervised multi-view image segmentation using radiance field propagation with bidirectional photometric loss to guide the reconstruction of semantic field. We design an EM-based mask refinement procedure to handle complex scenes. Our experiments show the effectiveness of approach, with useful applications on NeRF such as individual object rendering and editing.
Acknowledgements
We would like to thank Yichen Liu and Shengnan Liang for fruitful discussion the inception stage of the project. This research is supported in part by the Research Grant Council of the Hong Kong SAR under grant no. 16201420.
References
[1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 39(12):2481 22122995.. 2017.
[2] Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. In Advances Neural Information Processing Systems, 2019.
[3] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference Computer Vision, 2022.
10

---

[4] Liang- ChinggChhhhhhh Chen, George Papandreou. Florian Schroff,, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. In European Conference Computer Vision, 2018
[5] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision, 2018.
[6] Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In Advances in Neural Information Processing Systems, 2019.
[7] Ling-Ming Cheng. Guo-Xin Zhang, Niloy J. Mitra, Xialei Huang, and Shi-Min Hu. Global contrast based salient region detection. In The IEEE/ CVF Computer Vision and Pattern Recognition Conference, 2011.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2009.
[9] Sara Fridovich-Kar Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In International Conference on Computer Vision, pages 5501-5510, 2022.
[10] Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08500; 2020.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. International Conference on Computer Vision. pages 2961-2969, 2017.
[12] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017.
[13] Huaizu Jiang, Jingdong Wang, Zejian Yuan, Yang Wu, Nanning Zheng, and Shipeng Li. Salient object detection: A discriminative regional feature integration approach. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2013.
[14] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2017.
[15] Asako Kanezaki. Unsupervised image segmentation by backpropagation. In IEEE International Conference on Acoustics, Speech, and Signal Processing, 2018.
[16] Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture rich benchmark for unsuper- vised multi-object segmentation. arXiv preprint arXiv:2111. 2021.
[17] Wonjik Kim, Asako Kanezaki, and Masayuki Tanaka. Unsupervised learning of image segmentation based 2020.
differentiable feature clustering. IEEE Transactions on Image Processing,
[18] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: semantic object-aware neural scene representation. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference. 2022.
[19] Tsung- Yi Lin, Michael Maire, Serge J. Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Ziticck. Microsoft COCO: common objects in context. In European Conference on Computer Vision, 2014.
[20] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2015.
[21] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The conerete distribution: A continuous relaxation of discrete random variables. International Conference on Learning Representations, 2017.
[22] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[23] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
11

---

[24] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies primitives with multiresolution hash encoding. ACM Transactions on Graphics (TOG), 41(4)::02:111022155, July 2022.
[25] Michael Niemeyer and Andreas Geiger. Giraffe:
Representing scenes compositional generative neural feature fields. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2021.
[26] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In The Indian Conference Computer Vision, Graphics and Image Processing, 2008
[27] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural seene graphs for dynamic scenes. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference, pages 1856-28664 2021.
[28] Keunhong Park, Utkarsh Sinha. Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz. and Ricardo Computer Vision, 2021.
Nerfies: Deformable neural radiance fields. International Conference on
[29] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer, D-nerf: Neural radiance fields for dynamic scenes. In The /EEE/ CVF Computer Vision and Pattern Recognition Conference, pages 10318-10550 1011887.77, 2021.
[30] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision. pages 10901-10911 2021.
[31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 2015.
[32] O. Ronneberger. P.Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisseed Intervention, 2015.
[33] Pedro Savarese, Sunnie SY Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Information- theoretic segmentation by inpainting error maximization. In The IEEE/ CVF Computer Vision and Puttern Recognition Conference, pages 4029-4039, 2021.
[34] Selim Seferhekov, Vladimir Iglovikov, Alexander Buslaev, and Alexey Shvets. Feature pyramid network multi-class land segmentation. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pages 272-275, 2018.
[35] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. International Conference on Learning Representations, 2015.
[36] Cameron Smith, Hong-X Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B. Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition object light fields. In International Conference on Learning Representations, 2022.
[37] Karl Stelzner, Kristian Kersting, and Adam R Kosiorck. Decomposing 3d scenes into objects via unsuper- vised volume segmentation. arXiv preprint arXiv:2104.01 148, 2021.
[38] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
[39] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-201 1001, California Institute of Technology, 2011.
[40] Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Ya, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In The IEEE / CVF Computer Vision and Pattern Recognition Conference, pages 13524-13534, 2022.
[41] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In The IEEE/ TEE//CK CVF Computer Vision and Pattern Recognition Conference, 2015.
[42] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In The IEEE/ CVF Computer Vision and Pattern Recognition Conference, 2022.
[43] Bangbang Yang. Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object compositioon neural radiance field for editable scene rendering. In International Conference on Computer Vision, 2021.
12

---

[44] Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic- static bootstrapping. In The IEEE/ CVF Computer Vision and Puttern Recognition Conference, pages 2826-2836, 2021.
[45] Yanchao Yang, Antonio Loquercio. Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. The IEEE CVF Computer Vision and Pattern Recognition Conference, 2019.
[46] Alex Yu, Ruilong Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees real time rendering of neural radiance fields. In International Conference on Computer Vision, pages 2752-5761, 2021.
[47] Fisher Yu, Ari Seff,, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: arXiv:1506. 150560306633333 2015.
[48] Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu. Unsupervised Discovery of Object Radiance Fields. In International Conference on Learning Representations, 2021.
[49] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoin video using layered neural representation. ACM Transactions on Graphics (TOG), 40(4):1 2021.
[50] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In The JEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2017.
[51] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place labelling and understanding with implicit scene representation. International Conference on Computer Vision, pages 15838-15847, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [No] Not yet, but we
will make the code and data publicly available upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
Not yet, but will make the code and data publicly available upon acceptance.
(d) Did you discuss whether and how consent obtained from people whose data you're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "uNtf1hz9gs", "reviewer": "Reviewer_QA4f", "summary": "This paper focuses on unsupervised segmentation in 3D with multiview images. The main idea is to use a mixture model of multiple visually contrastive radiance fields to compositionally reconstruct the scene. The contrastiveness among the RFs is enforced by propagating average appearance to empty space and maximizing the visual differences between the reconstruction and the inversely mixed composition. Experiments show good results on a single-object real dataset and a multi-object synthetic dataset.", "questions": "It would be interesting to see if the same method works with single images. Have you ever tried that with 2D or 3D RFs?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n+ Interesting idea to create and use contrastiveness in compositional NeRFs for unsupervised segmentation.\n+ In general, the experimental results look good.\n+ Experiments on a real dataset.\n+ Comprehensive comparison to the state of the art.\n\nWeakness:\n\n- I feel a few aspects in technical description and experiments are not well clarified and justified. For Eq. (4), why does it need to be differentiable? It seems l(x) is only used in Eq. (5), but is discontinuous and not differentiable. Why are numbers in 2nd last row of Table 1 missing?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_7uOjLuMsB8", "reviewer": "Reviewer_fm5S", "summary": "The paper views image segmentation task from a 3D perspective, applies NeRF to the unsupervised multi-view image segmentation task for the first time. The paper guides the reconstruction of semantic field through the radiation field propagation, and proposes a mask refine procedure based on EM to handle complex scenes with multiple objects.", "questions": "The authors should explain why Propagation works intuitively?", "limitations": "When dealing with multi-object complex scenes, this method needs a strong feature extractor, which requires large-scale data for pretraining，It's hard to tell whether it's the strong feature extractor works or the paper‘s method works. Especially for the comparasion with other methods, the introduction of this large-scale external data will make the comparison unfair.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pos:\nThe method is novel. NeRF is appropriatly applied to deal with multi view image segmentation tasks from the perspective of 3D, and a bidirectional photometric loss is proposed to construct the unsupervised task. The experimental improvements are remarkable.\nThe overall writing of the paper is smooth and the overall structure is clear.\n\nNeg:\nLack of analysis on why propagation is useful makes it difficult to understand why it works.\nWhen dealing with multi-object complex scenes, this method needs a strong feature extractor, which requires large-scale data for pretraining，It's hard to tell whether it's the strong feature extractor works or the paper‘s method works. Especially for the comparasion with other methods, the introduction of this large-scale external data will make the comparison unfair.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "HQMvjfJhO7", "reviewer": "Reviewer_GXm6", "summary": "This paper propose an unsupervised multi-view object segmentation method leveraging on neural radiance field. The key motivation is that a good segmentation should  maximise the dis-similarity between content within each individual mask. Correspondingly, three key modules including radiance propagation, bi-directional photometric loss and EM-based post-processing are adopted to decompose a scene into different regions/objects. This paper presents promising qualitative results on object-centric scenes and synthetic multi-objects scenes under multiple observations. ", "questions": "\n-How does the system work on objects with rich textures or similar appearance to backgrounds? For example, BlenderMVS or DTU datasets? \nIn addition, under this situation, will the  propagation mechanism using average colour be still helpful?\n\n-How does the initial masks involve during training and how to balance different losses.\n\n-Is the system sensitive to pose registration accuracy?", "limitations": "The authors have addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well motivated and the pipeline design follows the idea of making prediction  of one part from others difficult.\n\n2. The evaluation and ablations are adequate on single-object scenes and multi-object scenes(synthetic)scenes. As far as I can tell this is the first multi-object segmentation framework in a purely unsupervised manner on complex scenes.\n\n3. Though there are some missing details (mentioned in the weakness part), the overall writing is good.\n\n\n\nWeaknesses\n\n\n1. It is not clear to me what equation (4) means and helps the optimisation? In addition, \\hat{l} is not mentioned in the remaining text.\n\n2. The object-level NeRF is driven to predict average colour within masks across the batch, the design choice seems a bit ad-hoc. \nWhat about predicting other related values to object regions like the maximum or minimum or other strategies? \nAs to objects with complex textures or various colours, I assume even the average colour may also not well represent the object appearance. Objects within the paper owns similar textures or colours, and I am not sure if the propagation is still feasible to other cases.\n\n\n3. Similar to previous points, the main concerns comes from the question, how much does the system rely on appearance instead of others like geometric cues. The objects within several experiment have a relatively clear difference in colour compared to backgrounds, which makes it a good \"colour segmenter\", limiting the pipeline towards complex scenes. \n\n\n4. From quantitative evaluations, the initial mask acts as an important role. Will the quality of the initial masks severely affect the final performance?  During training, will the initial segmentation loss be turned-off in the later phase of training to achieve better details?\n\n5. As an unsupervised method, how to properly choose the loss weighting in Eq.17?\n\n6. It would be good to demonstrate some illustrations about how each individual object-nerf works in and outside the object regions like Fig2, but for more complex scenes similar to Fig4 and Fig5.\n\n7. Is the system performance sensitive to errors within camera poses?\n\n8. Minor Issues.\n- L157. m_k(x)=0 --> m_k(x)=1\n- References should be added in L189 about the unsupervised single image-based approach, which is revealed late in the paper.\n- L207. I suggest adding references or some extra explanations here. Otherwise it is hard for readers to immediately know what is exactly thee deep feature extractor here and how it works.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0107dvh5kr", "reviewer": "Reviewer_1Do1", "summary": "This paper presents an approach to learn 3D object instance segmentation, given RGB images and camera poses. The main novelty is the \"propagated\" radiance field obtained by summing radiance field layers weighted by *inverted* masks. Correct labeling would maximize its photometric error. The network is trained using both traditional photometric loss and the proposed loss. EM-refinement is used to improve the result.", "questions": "Table 1 is confusing. It says “The best results without supervision are boldfaced.” But it is not clear on its own which rows they are. If I understand correctly, ablated results perform worse, as expected. And all the other papers had some label or bounding box supervision.  It might be helpful to indicate which additional supervisory signals they used, in each row.\n\nWas this baseline considered: train vanilla NeRF, do 3D reconstruction (e.g. marching cubes), treat each connected volumetric component as an instance (assume known floor level). It would also be unsupervised.\n\nL157, is m_k(x) = 0 supposed to be m_k(x) = 1 ?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: Unlike prior work, instance segmentation is possible. And inverting the mask is an interesting idea.\n\nWeakness: The scenes do not seem complex enough for the task. An object close to a wall (in addition to the floor) or another object may be difficult to segment. It would be insightful to see some failure cases (either real or synthetic).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept