Unsupervised 3D Scene Representation Learning via Movable Object Inference
Authors: Honglin Chen
Wanhee Lee
Hong-Xing Yu
Rahul Mysore Venkatesh
Joshua B. Tenenbaum
Daniel Bear
Jiajun Wu
Daniel LK Yamins
Venue: Submitted to ICLR 2023 (2023)
Keywords: 3D representation learning
self-supervised learning
object-discovery
neural rendering

Abstract: Unsupervised, category-agnostic, object-centric 3D representation learning for complex scenes remains an open problem in computer vision. While a few recent methods can now discover 3D object radiance fields from a single image without supervision, they are limited to simplistic scenes with objects of a single category, often with a uniform color. This is because they discover objects purely based on appearance cues—objects are made of pixels that look alike. In this work, we propose Movable Object Radiance Fields (MORF), aiming at scaling to complex scenes with diverse categories of objects. Inspired by cognitive science of object learning in babies, MORF learns 3D object representations via movable object inference. During training, MORF first obtains 2D masks of movable objects via a self-supervised movable object segmentation method; it then bridges the gap to 3D object representations via conditional neural rendering in multiple views. During testing, MORF can discover, reconstruct, and move unseen objects from novel categories, all from a single image. Experiments show that MORF extracts accurate object geometry and supports realistic object and scene reconstruction and editing, significantly outperforming the state-of-the-art.

Under review as a conference paper at ICLR 2023
UNSUPERVISED 3D SCENE REPRESENTATION LEARN- ING VIA MOVABLE OBJECT INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised, category-agnostic, object-centric 3D representation learning for complex scenes remains an open problem in computer vision. While a few recent methods can now discover 3D object radiance fields from a single image without supervision, they are limited simplistic scenes with objects of single category, often with uniform color. This is because they discover objects purely based on appearance cues-objects are made of pixels that look alike. In this work, we propose Movable Object Radiance Fields (MORF), aiming at scaling to complex scenes with diverse categories of objects. Inspired by cogmitive science studies of object learning in babies, MORF learns 3D object representations via movable object inference. During training, MORF first obtains 2D masks of movable objects via self-supervised movable object segmentation method; it then bridges the gap to 3D object representations via conditional neural rendering in multiple views. During testing, MORF can discover, reconstruct, and move unseen objects from novel categories, all from single image. Experiments show that MORF extracts accurate object geometry and supports realistic object and scene reconstruction and editing, significantly outperforming the state-of titt--of-thearrt
INTRODUCTION
Learning object-centric 3D representations of complex scenes is critical precursor to a wide range of application domains in vision, robotics, and graphics. The ability factorize a scene into objects provides the flexibility of querying the properties of individual objects, which greatly facilitates downstream tasks such as visual reasoning, visual dynamics prediction, manipulation, and scene editing. Furthermore, we hypothesize that building factorized representations provides strong inductive bias for compositional generalization (Greff et al. 2020), which in turn enables the model to understand novel scenes with previously unseen objects and configurations.
While supervised learning methods have shown promise learning 3D object representations (such
neural radiance fields (Mildenhall et al. 2020) from images (Ost et al. 202 Kundu et al. 2022
Müller et al.] 2022). ) they rely on annotations of specific object and scene categories. A recent line of work Yu et al. 2022 Stelzner al. 2021) has explored the problem of unsupervised discovery of object radiance fields. These models can be trained from multi-view RGB or RGB-D images to learn object-centric 3D scene decomposition without annotations of object segments and categories. However, they are only demonstrated to work well on simplistic scenes. It remains challenging to obtain accurate reconstructions on more complex datasets. A fundamental reason that they heavily rely on visual appearance similarity to discover object entities, which limits their scalability beyond simple texture-less objects.
In this work, we aim to scale unsupervised 3D object-centric representation learning to complex visual scenes with textured objects from diverse categories. To this end, we propose a Movable Object Radiance Fields (MORF) model, which learns to infer 3D object radiance fields from single image. Rather than appearance similarity, the underlying principle MORF uses to discover object entities is material coherence under everyday physical actions, i.e., an object is movable space (pelke 1990). However, it is challenging to obtain learning signals to directly infer movable objects in 3D. MORF addresses this problem by integrating a recent self-supervised 2D movable object segmentation method, EISEN (Chen et al., 2022). to extract movable object segments in 2D images, well as differentiable neural rendering bridge the gap between 2D learning signals and
whole in 3D

---

Under review as a conference paper at ICLR 2023
Inpar
Object rediance felds tasi and geometry (bottomi
Top. input view, bottom nondl View
Illustration of unsupervised, category-agnostic. object-centric 3D representation learning. Given a single image, our goal is to infer object radiance fields that allow photometric and geometric 3D reconstruction. This factorized representation enables 3D seene manipulation, including moving object and replacing the background.
Figure I:
3D inference. MORF learns to conditionally infer object radiance fields from the segmented images, which provide strong inductive bias for object-centric factorization 3D scenes.
Specifically, we pretrain EISEN on optical flow from unlabeled videos. EISEN learns object image segmentations by perceptually grouping parts of scene that would move as cohesive wholes, serving as module that estimates high-quality object segmentations MORF learns to extract object-centric latent representations from segmented images and generate object radiance fields from the factorized latents. To facilitate high-quality reconstruction of textured objects, our latent object representation consists of both an entity-level latent and pixel-level latent that better encodes appearance details.
static images. After pretraining,
To evaluate our method, we propose a challenging dataset with a diverse set realistic-looking objects, going beyond simplistic scenes considered by current unsupervised 3D object discovery methods (Yu et al., 2022, Stelzner el al. 202T; Sajjadi et al., 2022b). We demonstrate that MORF can learn high-quality 3D object-centric representations complex visual scenes, allowing photometric and geometrie reconstruction for these scenes from single views (Figure D Moreover, our learned representations enable 3D scene manipulation tasks such moving, rotating, and replacing objects, and changing the background of complex scenes. Beyond systematic generalization to unseen spatial layouts and arrangements, we further show that MORF is able generalize to unseen object categories and appearances while maintaining reasonable reconstruction and geometry estimation quality.
In summary, our contributions are three-fold. First, we propose scaling the learning of unsupervised, category-agnostic, object-centric 3D representation learning beyond simplistic scenes by discovering objects with coherent motion, in addition to visual appearance. Second, to instantiate propose Movable Object Radiance Fields (MORF), which integrates 2D movable object segmentation with neural rendering to allow 3D movable object discovery. Third, demonstrate that MORF allows photometric and geometric reconstruction and editing complex 3D scenes with textured objects from diverse unseen categories.
idea,
RELATED WORK
Unsupervised 2D object discovery Our method is closely related to recent work on unsupervised scene decomposition, which aims decompose multi-object scenes into separate object-centric representations from images without human annotations. Most works formulate the problem as
learning compositional generative models the 2D image space. They decompose a visual scene
into a set of localized object-centric patches (Eslami et al. 2016] Crawford & Pincau] 2019 Kosiorek
al.
2018:
Lin et al.
2020 Jiang al. 2019a) or set of scene mixture components (Burgess
et al.
2019
Greff et al.
2019 2016 2017 Engelcke et al. 2019 Locatello et al., 2020) Monnier
ct al.
2021
Jiang et al.
2019b). The scene mixture models typically generate single-object RGBA
images and blend them reconstruct the full-scene images using iterative inference with recurrent
networks (Burgess et al. 2019) or set-based convolutional encoders (Locatello et al., 2020). However,
these methods have far been unable to scale to complex real-world images. A recent branch of
work on self-supervised object segmentations explores additional supervision signals such as motions
and depth for learning object segmentations (Bear al. 2020
Kipf al. 2021: Chen al. 2022
2

---

Under review as a conference paper at ICLR 2023
Bao et al. 2022 Elsayed et al.) 2022 Ye et al. the 3D nature of scenes, and thus they do not provide 3D understanding of the underlying scenes.
2022). However, these 2D methods not aware of
Unsupervised 3D object discovery. Discovering objects from image collections has been long- standing topic in computer vision, but earlier works on object discovery k.. co-segmentation) represent objects as 2D image segments without 3D information (Russell et al. 2006 Sivie et al. 2005, 2008 Grauman & Darrell 2006
Joulin et al. 2010 Rubio et al. 2012 Vicente et al.) 2011 Rubinstein al. 2013 Cho et 2015 Li et al. 2019 Vo al. 2020). Recently, some works have been focusing on discovering 3D object representations. A related branch of works focuses on 3D reconstruction from a single image Ye et al. 2021 Kulkarni et all 2019 2020, Kanazawa et al. shape priors, making it difficult to scale to complex real-world data. Elich et al. (220)) infer object shapes (Park et al., 2019) from a single scene image. [Chen et al. (2020) extend Generative Query Network (Eslami et al.) 2018) to decompose 3D scenes. Notably, the closest to our work is a recent branch of works that focuses on inferring 3D neural object representations from single images (Yu et al. 2022] Smith et al. 2022 Stelzner et al., 2021 or sparse views (Sajjadi et al. 20226). However, these methods rely on visual appearance to discover object entities. This fundamental assumption makes them difficult to scale to complex scenes with textured objects, diverse object categories, or objects under different lighting. In contrast, our approach leverages motions as the underlying object concept, which is category-agnostie and generalizable to different object appearances.
2018: Wu et al.) ( However. it requires strong category
Scene de-rendering. Aiming to provide full scene understanding, a line of scene de-rendering works have shown reconstructing 3D object-centric representations in specific types of scenes (Wu
al.
2017
Yao et al. 2018 Kundu al.
2018
Ost et al., 202T Kundu al., 2022 Yang et al.
2021
Wu et al., 2022). Recently, Ost et al.
2021) propose to represent dynamic scenes into a scene
graph where each node encodes object-centric information. Müller et al. (022) recover 3D object information such as shape, appearance, and pose in autonomous driving scenes. However, these methods rely on manual annotations of object categories (such as cars) and scene categories (such as street scenes). Similarly, Gkioxari et al. (2022) propose method that learns to predict 3D shape and layout for objects by relying on 2D bounding box supervision. Our approach only requires object motion in video for inferring object segmentations during training, without requiring manual annotations.
Neural scene representations and rendering. Our object representation is based on recent pro- gresses in neural scene representations (Park et al. 2019: Mescheder et al. 2019; Sitzmann et al. 2019) and neural rendering (Tewari et al.] 2020). Neural scene representations implicitly model 31> scenes using the parameters of deep networks, which could be learned from only 2D images (Niemeyer Sitzzaan et al. 2019) with differentiable rendering techniques (Kato et al. 2020 Tewari
al. 2020
et al. 2020 Specifically, Neural Radiance Fields (NeRFs) (Mildenhall et al. 2020) has shown photorealistic scene modeling of static scenes using only images. The most relevant works in this line aim to infer NeRFs from single image (Yu et al. 2020) Kosiorek et al.) 202 Rematas et al. 1021). While these works focus on single objects or holistic scenes, we address decomposing a multi-object scene without human supervision. Another relevant branch of works aims at incorporating NeRFs into compositional models (Niemeyer & Geiger 2020 202T). such as Niemmyyr & Geiger (020).
While they target at scene synthesis, instead focus on multi-object inference, which GIRAFFE cannot address (Yu al. 2022).
3 MOVABLE OBJECT RADIANCE FIELDS (MORF)
We now describe the problem formulation and our approach, Movable Object Radiance Fields. Given a single input image of scene that potentially has objects from diverse categories, our goal is to factorize the scene into set of object-centric conditional radiance field representations. To allow unsupervised decomposition of such complex 3D scenes, we propose learning to discover movable objects by integrating 2D movable object inference with 3D-to-2D neural rendering.
Movable Object Radiance Fields therefore has three stages. First, it decomposes the input image by inferring its 2D object segmentation masks (Figure 2.. The segmentation extraction network is trained separately and self-supervised by optical flow for grouping scene elements that often move together into an object segment. Second, for each mask, Movable Object Radiance Fields learns an object radiance field with object and pixel latent codes for object features and locally varying details, respectively (Figure 2b). Finally, these object radiance fields are then composed to re-render the
3

---

Under review as a conference paper at ICLR 2023
Background
Training
Inference
Beaggaudd
BG
Movil Adlees
EISEN
reice:
Novel veew
Zirapo
Daurstry
On Jlaa
88
wdil og
(a)20 decomposition
(b) Litting to 3D
(c) Compositional rendering
Figure 2: Illustration of Movable Object Radiance Fields (MORF). MORF takes as input a single image of scene with potentially diverse objects, and infers 3D object and background radiance fields. (a) MORF integrates image-basee movable object inference method, EISEN, that predicts create masked images as inputs for learning object radiance fields; (b) MORF generates object radiance fields conditioned on the latent object and pixel codes. (c) During training, MORF reconstructs the novel view via compositional rendering and is supervised by reconstruction losses. During inference, MORF takes a single view of new scene, and infers object and background radiance fields in single forward pass.
of object masks, which used
scene from multiple views, supervised by the reconstruction loss (Figure 2 We now describe each component in detail.
3.1 MOVABLE OBJECT INFERENCE IN 2D
Given a single RGB image, we first compute a 2D object segmentation mask, represented as H x Wx x K binary tensor M where is the number of object masks. The background mask MD
can be computed by taking the complement of the object masks union: M = M
We adopt the EISEN architecture from Chen al. (2022) for generating high-quality object segmen- tation masks. The core idea of EISEN is to construct high-dimensional plateau map representation (of shape H' x W x Q) for each image, in which all the feature vectors @i belonging to the same object are aligned (i.e., have cosine similarity > 1) and all feature vectors that belong to other objects are nearly orthogonal (cosine similarity ss 0). Given this representation, the object segments can be easily extracted from a plateau map by finding clusters of vectors pointing in similar directions.
More specifically, EISEN first applies convolutional backbone on the input image to obtain a feature grid, followed by affinity prediction module that computes pairwise affinities between features pixel location (i and the features at its neighboring locations within a local window. Then EISEN constructs a graph, with nodes represented by Q-dimensional feature vectors and edges represented by the pairwise affinities. A message- saasaaggpassinggggg graph neural network is run on the graph to construct the plateau map representations, by passing excitatory messages that align the feature vectors of nodes belonging to the same object and inhibitory messages that orthogonalize the feature vectors of nodes belonging to distinct objects. Once the plateau map representation is obtained, EISEN imposes winner-take-all dynamics on the plateau map to extract object segments. We refer the readers to Chen
al. 2202)) for more implementation details EISEN.
During training, EISEN learns the pairwise affinities from optical flow estimates from a RAFT TTed
Deng 2020p network pretrained on Sintel (Mayer et al. 2016). Consider pair of scene elements (a, b) that project into image coordinates (i.j) and (7", respectively. If only one is moving, it usually the case that they do not belong to the same object; when neither is moving, there is no
information about their connectivity. We use this physical logic to construct pairwise supervision signal for EISEN's affinity matrix. The EISEN training loss is the masked row-wise KL divergence between the predicted and target connectivity. Although EISEN requires frame pair input for computing optical flow during training, only requires a single static image during inference time for computing the object segmentation masks, and subsequently predicting the object radiance fields.
4

---

Under review as a conference paper at ICLR 2023
3.2 LEARNING OBJECT RADIANCE FIELDS IN 3D
We model the 3D representations of objects and backgrounds as conditional neural radiance fields. Given input image and its predicted EISEN masks, we compute both global slot-based conditioning S E E and local pixel conditioning p E Ry for high-fidelity reconstruction of complex scenes with locally varying appearances. Since the geometry of the background and foreground objects are highly different, representing them using the same conditional NeRF might impede the system's 's capacity to model complex and diverse object geometries. Therefore, we parameterize the background NeRFs and object NeRFs using two separate conditional MLPs with parameters O, and O.. respectively. The MLP parameters of the object NeRFs shared across all the foreground objects. The latent codes are used as input to the MLP networks, along with 3D position encoding 7(x) and view directions d.
(x), dls S. .P') = (oo, Cb), for (1x)), dls' pp) co), iE fO....., K}
(1)
(2)
To compute object latent codes, we adapt the slot-based update mechanism proposed by Locatello et al. (2020) for inferring object latents. Unlike the original formulation of uORF, which computes object slots using an attention module, we directly compute the initial object latents via average pooling of a convolutional feature map using the predicted EISEN segmentation masks s² W e (feat,
where W = /
1= M)). The latent codes are iteratively updated via a learnable Gated Recurrent Unit (Chung et al. 2014), s + G updates"). producing the final latents for conditional neural radiance field. The latent codes s for the foreground objects are computed using the same formulation.
To obtain the pixel latent codes, we first mask the input image using M and M and extract a convolution feature grid for the background and each object respectively W (I).
W (T)) For each query point x on camera ray, we follow Yu et al. (2020) to retrieve the image features of each object by projecting onto the image plane coordinates т (x Jand extract latent codes from the feature grid via bilinear interpolation, obtaining pixel latent codes p³ W
and p² W (T))))
3.3
COMPOSITIONAL RENDERING
At each point in the rendering of a scene, the final pixel value is a combination of contributions from each individual scene element the 3D volumes projecting to that point. We follow uORF Yu et al. 2022) and take weighted average of the individual components to obtain the combined density and color e. The composite volumetric radiance field is then rendered into a 2D image via the numerical integration of volume rendering by sampling S discrete points along each pixel ray parameterized as r()) = + Id, with the ray origin o and ray unit direction vector d. The points on each ray are sampled between pre-specified depth bounds [t,, ty], with distance d5 between adjacent samples along the ray. Thus, the final pixel color is given by:
C(r) = Σ i=0
Till -xx(eexp(-a exp(-o;0i)]c,,
T= = =exp((- - ~===0 ( o,Oi),
(3)
where
o = E,,00 c= <=E,pseh prCs, p== =
of J__ Ex
j.k E K}.
(4)
3.4
LOSS FUNCTION
During training, randomly select single image of a scene as input, and render multiple novel views. We train the model using both reconstruction loss and perceptual loss: L = L, + Xp with Ap = 0.006. The reconstruction loss is the L2 loss between the rendered image and the ground-truth image L. = || - Í||². Since reconstruction loss is sensitive to small geometric imperfections and often results in blurry reconstructions, especially due to uncertainties in rendering novel view, we add a perceptual loss term to mitigate this problem. We compute the perceptual loss as C, (I) ex (f) 1| where (+)() is k-th layer of an off-the-shelf VGG16 (Smonnyan & Zisserman 2014 image encoder with frozen pre-trained weights.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We demonstrate our approach on three tasks: (a) novel view synthesis, (b) scene geometry recon- struction, and (c) editing scenes by moving objects, replacing objects, and changing the background.
Datasets We generated three variants of complex synthetic scene dataset using the World simulation environment (Gian et al., 2020). Each scene includes four camera views with a random azimuth angle and a fixed elevation; the camera always points at the scene center.
Playroom dataset contains a wide range of realistically simulated and rendered objects. Each scene includes 3 objects randomly sampled from set of 2000 object models. These models are drawn from wide set of categories and have range of complex 3D shapes and textures. They are placed in random positions and poses in rooms selected randomly from a collection of indoor environments with varying 3D room layouts and floor/wall textures. In each scene, one object is invisibly pushed to generate object motion. There are 15,000 scenes for training and 600 for testing.
Playroom-noo novel dataset contains novel object models for evaluating the generalization performance of the models. Each scene contains 3 objects randomly sampled from 100 distinct object models that held out from the Playroom scenes. They are placed in the same room environments seen in the Playroom dataset. There are 600 scenes for evaluation.
Playroom-edit dataset is designed for evaluating model's ability to manipulate object radiance fields and synthesize novel images. The dataset contains scenes that result from three types of editing: moving objects, replacing objects, and changing the background. For object moving, we randomly pick one object in the scene and teleport it to a random position. For object replacement, we switch a randomly selected object with an object from a different scene. For background replacement, we similarly switch the background with that of another scene. For each scene editing task, we render 200 test scenes for evaluation.
Baselines We compare MORF to the slot-conditioned unsupervised object radiance field method uORF YYu et al.] 2022) and nonfactorized pixel-conditione method pixeINeRF (Yu et al. 2020) Both methods learn radiance fields from RGB images without ground-trute object annotations or depth supervision. We also compare to an ablated version of MORF trained without pixel latents to illustrate the benefits of local conditioning in 3D representation learning. We adopt the same training procedures and hyperparameter choices as reported in the original papers. For a fair comparison, all models receive input images with the same resolution and are trained with the same batch size.
4.1 NOVEL VIEW SYNTIEESIS
We randomly select one camera view of each scene as input and use the remaining three images as ground truth for evaluating the quality of novel view synthesis. Besides uORF and RiXe we also compare with the pixel feature-ablate version of MORF. All the models are evaluated using the standard image quality metrics PNSR, SSIM (Wang et al. 2004). and LPIPS (Zhang et al. 2018).
Results As shown in Table M Figure 3 MORF outperforms the baseline methods both quan- titatively and qualitatively. uORF and SKT Sajjadi et al.] 2022a) are able to learn rough object decompositions and position estimates, but fail to represent the object shapes accurately, resulting in inaccurate rendering from novel views. The version of MORF without pixel features as condition- ing performs second-best. Although both methods object latent conditioning, the latter attains substantially better reconstructions, suggesting accurate object segmentations help constrain the optimization of the neural implicit function. Both MORF and PixeINeRF are able to render novel views reasonably well. We highlight that PixeINeRF's reconstructions are blurry both in background regions (such as floor tiles) and on objects (such the dumbbell), while MORF's reconstructions are sharper. This demonstrates the importance of accurate object decomposition on high-quality novel view synthesis. The comparison to the pixel natture-ablated MORF illustrates the advantage of using pixel features over object latents in capturing fine-grained details of scenes.
4.2 GEOMETRY RECONSTRUCTION
To evaluate the quality of 3D scene representations, we first extract meshes from each model's learned density field with the marching cubes algorithm (Lewiner et al. 2003). We compute the density field by evaluating the foreground decoder at grid points in the world coordinate system. For a fair comparison, all the models are evaluated using the same grid size of 256.

---

Under review as a conference paper at ICLR 2023
Playroom
Playroom-novel
Input view
uORF
PixeINeRF
SRT
MORF
MORF
Novel view (GT)
Figure 3: Qualitative results novel view synthesis the Playrom and Playmom-novel datasets. The objects Playroom novell datasets are not included the training set. MORF outperforms the baseline models on both foreground and background reconstruction. MORF is better at reconstructing fine fine-graing grained textures and object geometries than the other models.
Playroom
Playroom-novel
View synthesis
Goometry
View synthesis
Geometry
Models
I.IPSS
SSIMF
PSNRT
O-CD
J.
I.PISS..
SSIM
PSNR1
S-CD)
WORF
0.348
0.634
21.5
0.324
0.113
0.350
0.636
21.6
0.324
0.150
piseINeRF
0,250
0.745
24.4
0.133
0.265
0.725
23.1
0.128
SRT
0.352
0.704
21.3
0.355
0.704
21.3
MORF pix)
0,264
0.122
MORE (ours)
0.161
0.784
24.5
0,208
0.078
0.189
0.755
22.9
0.224
0.110
Table Quantitative comparison on novel view synthesis and geometry reconstruction. Novel view synthesis performance is measured by LPIPS, SSIM, and PSNR. Geometry reconstruction is measured by Object Chamfer Distance (O-CD) and Scene Chamfer Distance (S-CD). O-CD mcasures the geometry reconstruction quality of individual object meshes. S-CD measures the 3D layout of multiple objects in the seene by comparing the reconstructed and ground-truth scene meshes. For pixelNeRF (Yu et 2020), O-CD not reported due to the lack object decomposition.
Metrics We compare the MORF and other baselines using Chamfer Distance (CD) (Sun et al. 2018). We compute two types of CDs. object mesh Chamfer Distance (O-CD) and scene mesh Chamfer Distance (S-CD). The O-CD metric focuses on measuring the quality of each individual mesh reconstruction, while the S-CD metric focuses on measuring the quality of the objects' layout in 3D. We compute S-CD on the foreground object meshes in scene. Points are uniformly sampled on the mesh surface to create a dense point cloud; then points are randomly sampled from point cloud, where N is 1,024 per object mesh and 3,072 per scene mesh. We normalize the point cloud coordinates into a unit cube before CD calculation. Due to the lack of object dccomposition, pixelNeRF only outputs single mesh encompassing both the foreground objects and the background. For a fair comparison of S-CD with MORF, we remove the background meshes by setting the density of the grid points below z-value threshold zero before applying marching cubes. We search for the threshold with the best S-CD on a validation set, and use the threshold to calculate pixeINeRI's S-CD on the Playroom and Playroom-nove test datasets.
Results We show the results Table and Figure MORF outperforms all methods in terms of O-CD and S-CD. As seen in Figure uORF Yu et al. 2022) only able recover the coarse

---

Under review as a conference paper at ICLR 2023
Playroom
Playroom-novel
Input view
uORF
PixeINeRF
MORF
MORF
(wlo pixel feat.)
(with GT masks)
Figure 4: Qualitative results on 3D segmentation and mesh reconstruction. MORF produces more accurate mesh reconstructions than uORF and pixeINeRF, well the ablated model of MORF trained without pixel features conditioning. PixeINeRF only outputs single mesh encompassing both the foreground objects and background. We remove background mesh pixeINeRF and visualize foreground objects only.
Move object
Replace object
Replace background
Models
LPIPS
SSIM
PSNR
LPIPS
SSIM
PSNRT
LPIPS
SSIM
PSNR
uORF
0.381
0.573
20.7
0.384
0.575
20.8
0.371
0.593
20.4
MORF (no pix)
0.302
0.706
21.9
0.328
0.701
21.4
0.319
0.700
20.9
MORF (ours)
0.223
0.758
22.7
0.250
0.751
22.1
0.232
0.758
21.7
Table 2: Quantitative comparison on three scene editing tasks. MORF outperforms uORF on the metrics.
geometry of the objects. pixelNeRF Yu et al., 2020) tends to miss the fine details of small objects and thin objects, while MORF is able learn more fine-grained object geometry given single image.
4.3 SCENE EDITING
We consider three scene editing tasks: moving objects, replacing objects, and replacing the back- ground. For moving and replacing objects, we follow the protocol from the uORF work (Yu et al. 2022) and select the object that has the largest IoU with the ground-truth masks of the target object for editing. piXelNeRF (Yu et al. 2020) is not comparable on scene editing due to the lack object decomposition. We report LPIPS, SSIM and PSNR on the Playroom-edl edit dataset. Ground-tree -ruth masks are only used for selecting which object to edit. EISEN segmentation masks from a pretrained model used in the feedforward pass of MORF all the images. We show the results in Table and Figure 55] MORF outperforms uORF on all metrics across the three editing tasks. MORF trained without pixel features results blurry reconstruction on some of the objects.
4.4 ABLATION STUDIES
MORF performance on Playroom drops slightly when object latents are not used, though improves slightly on the O-CD metric. In contrast, pixel latents are crucial for MORF perform well on both novel view synthesis and geometry reconstruction. This implies that although object-centricity provides strong optimization constraint for learning object radiance fields, is insufficient for modeling the fine details of object textures and shapes; local pixel features are critical for scaling to complex scenes.
8

---

Under review as a conference paper at ICLR 2023
A. Move object
Edited scene (Input view)
Edited scene (novel view)
Input image
B. Replace object
Edited scene (input view)
Editdd scene (navel view)
Input image
Replace background
Editdd scene (Input view)
Edited scene (novel view)
Input image
GT
OORF
MORF (wl pixel feat.)
MORF
Figure 5: Qualitative results three and novel views are shown the top bottom rows respectively. MORF is able manipulate individual object radiance fields to generate novel scenes. Both uORF and MORF (without pixel features) are able to perform scene editing to limited extent, but their reconstructions are blurry. This comparison shows the advantage of using both EISEN segmentations and pixel latents for scene editing tasks.
editing tasks. The synthesis of the edited
from both input
Finally, we compare MORF trained with EISEN segmentation masks MORF trained with GT masks. Unsurprisingly. the latter shows better performance all metries. Qualitatively, we observe that MORF occasionally misses small objects or object parts, which is likely a direct consequence of EISEN failing accurately seg- ment these fine-scale scene elements. This indi- cates that improving unsupervised 2D segmen- tation should lead to further improvement of MORF's 3D representation learning.
Mask types
slot pixel leat.
Playroom
leat.
IPIPS
SSIM
PSNR O-CD J S-CD
0.266
EISEN EISEN GT
x
0.170
0.776
24.3
0.198
0.079
0.162
0.781
24.4
0.208
0.080
/
0.140
0.791
25.0
0.147
0.036
Table 3: Ablation study results.
CONCLUSION
In this work, we propose the Movable Object Radiance Fields (MORF) model, which scales unsuper- vised 3D object-centric scene representation learning to complex and diverse multi-object scenes. We demonstrate that MORF enables faithful photometric and geometric reconstruction of scenes with unseen configurations from a single view, generalizes well to unseen object categories, and supports complex editing tasks. We believe positive results suggest the promise of further scaling unsupervised 3D factorized representation learning to more complex scenes.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we will release the training and testing code, as well as the data to reproduce our results upon publication.
REFERENCES
Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discovering objects that can move. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1 11789-11798, 2022.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Josh Tenenbaum, et al. Learning physical graph representations from visual scenes. Advances in Neural Information Processing Systems, 33, 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 13390, 2019.
Chang Chen, Fei Deng, and Sungjin Ahn. Learning infer 3d object models from images. arXiv preprint arXiv:2006.061100 2020.
Honglin Chen, Rahul Venkatesh, Yoni Friedman, Jiajun Wu, Joshua B Tenenbaum, Daniel L.K Yamins, and Daniel M Bear. Unsupervised segmentation in real-world images via spelke object inference. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1201-1210, 2015.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Cathrin Elich, Martin R Oswald, Marc Pollefeys, and Jörg Stückler. Semi-supervised learning of multi-object 3d scene representations. arXiv preprint arXiv:2010.04030, 2020.
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprint wwxvv:226.00766. 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13052 2019.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukeuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint arXiv: 1603. 006755 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 2018.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threcdworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.0495. 2020.
Georgia Gkioxari, Nikhila Ravi, and Justin Johnson. Learning 3d object shape and layout without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1695-1704, 2022.
10

---

Under review as a conference paper at ICLR 2023
Kristen Grauman and Trevor Darrell. Unsupervised learning of categories from sets of partially matching image features. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), 2006.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Jürgen Schmidhuber, and Harri Valpola. Tagger: Deep unsupervised perceptual grouping. arXiv preprint wxXiv:6666066244 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization.
arXiv preprint arXiv: 03498, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.0526. 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. Scalable object-oriented sequential generative models. Unknown Journal, 2019a.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv:1910.02384, 2019b.
Armand Joulin, Francis Bach, and Jean Ponce. Discriminative clustering for image co-segmentation. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
Angjoo Kanazawa. Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386, 2018.
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey, arXiv preprint arXiv:2006.12057 2020.
Thomas Kipf, Gamalcldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:21 11,12944 2021.
Adam R Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. arXiv preprint arXiv: 1806.0) 1606.017944 2018
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo Rezende. Nerf-vae: A geometry aware 3d scene generative model. arXiv preprint arXiv:2104.00587 2021.
Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 2202-2211, 2019.
Nilesh Kulkarni, Abhinav Gupta, David Fouhey, and Shubham Tulsiani. Articulation-wwaree canonical surface mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 452-461. 2020.
Abhijit Kundu, Yin Li, and James M Rehg. 3d-renn: Instance-level 3d object reconstruction via render-and-compare. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. arXiv preprint arXiv:2205.0433 043304 2022.
Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira, and Geovan Tavares. Efficient implementation of marching cubes' cases with topological guarantees. Journal of graphics tools, 8(2):1155, 2003.
11

---

Under review as a conference paper at ICLR 2023
Bo Li, Zhengxing Sun, Qian Li, Yunjie Wu, and Anqi Hu. Group-wist deep object co-segmentation with attteettion recurrent neural network. In Proceedings of the IEEE/CVF International Confer- 2019.
ence on Computer Vision, pp.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprini arXiv:2001.024077 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint Xiv:2000 5005,, 2020.
Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4040-4048, 2016.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint 000::2003.08934, 2020.
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry. Unsupervised layered image decom- position into object prototypes. arXiv preprint arXiv:2 14575, 2021.
Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Autorf: Learning 3d object radiance fields from single view observations. arXiv preprint arXiv:2204.03553, 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. arXiv preprint arXiv:2011.12100 2020.
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance fields. arXiv preprint arXiv:2103.17269 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic seenes. In Proceedings of the IEEE conference on computer vision and pallern recognition, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Konstantinos Rematas, Ricardo Martin-Brualla and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from a single view. arXiv preprint 102.:::002.08600, 2021.
Michael Rubinstein, Armand Joulin, Johannes Kopf, and Ce Liu. Unsupervised joint object discovery and segmentation internet images. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2013.
Jose C Rubio, Joan Serrat, Antonio López, and Nikos Paragios. Unsupervised co-segmentation through region matching. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Bryan C Russell, William T Freeman, Alexei A Efros, Josef Sivie, and Andrew Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 06), 2006.
12

---

Under review as a conference paper at ICLR 2023
Mchdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Latent Scene Representations. CVPR, 2022a. URL https://
srt-paper. tiw....................... io/
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. arXiv preprint wwXiv:2206.06922, 2022b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous ad-strructre-aware neural scene representations. In Advances in Neural Information Processing Systems, pp. 1111-1132, 2019.
Josef Sivie, Bryan C Russell, Alexei A Efros, Andrew Zisserman, and William T Freeman. Discover- ing objects and their location in images. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 2005.
Josef Sivie, Bryan Russell, Andrew Zisserman, William T Freeman, and Alexei Efros. Unsuper- vised discovery of visual object class hierarchies. In 2008 IEEE Conference Computer Vision and Pattern Recognition, 2008.
Cameron Smith, Hong- Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint 2rXiv:2205.039233 2022.
Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01 148, 2021.
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for dingle-image 3d shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Bruall Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art neural rendering. Computer Graphics Forum, 792))701 2020.
Sara Vicente, Carsten Rother, and Vladimir Kolmogorov. Object cosegmentation. In CVPR 201 2011.
Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. European Conference on Computer Vision, pp. 779-795. Springer, 2020.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Object-composttinaal neural implicit surfaces. arXiv preprint arXiv:2207.09688 2022.
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. arXiv preprint arXiv:2107. 10844, 2021.
13

---

Under review as a conference paper at ICLR 2023
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning conposstoom neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13779-13788, 2021.
Shunyu Yao, Tzu Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T Freeman, and Joshua Tenenbaum. 3d-aware scene manipulation via inverse graphies. arXiv preprint arXiv:1808.09351 2018.
Vickie Ye, Zhenggi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites unsupervised video decomposition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2657-2666, 2022.
Yufei Ye, Shubham Tulsiani, and Abhinav Gupta. Shelf-supervised mesh prediction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8843-8852 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. arXiv preprint arXiv:2012.02190, 2020.
Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
14

---

Under review as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIAL
6.1 PLAYROOM DATASET
The Playroom dataset is generated with ThreeDWorld (Gan et al. 2020) using custom code. The dataset contains 15,000 training scenes and 600 test scenes. Each scene includes 3 objects randomly sampled from set of unique 2,086 object models in 231 object categories. The objects are rendered photo-realistically with complex geometry and textures. Table I and Figure show the comparison between the Playroom dataset and the MultiShpeeNet datasets.
ThreeDWorld - Playroom
MultiShapeNet- Easy
MuttiShapeNet Hard
Figure 6: Qualitative comparison between Playroom, MultiShapeNNt--ang and MultiShapeNet Hard
We simulate 2 video frames for each scene, with randomly selected ob- ject invisibly pushed the first frame to generate object motion. At each time step, we render views at resolu- tion 128 with random camera azimuth angles. is important to note that we only use the video frame pairs for training EISEN. We only the static images single time step for learning radiance fields. Figure 7 shows examples of the video frame pairs from the dataset, along with the BISEN segmentations and ground truth segmentation masks.
Dataset
scenes
objects
categories
cav. maps
obj. per
views
scene
Playroom
2k
231
3
MSN-Easy
80k
12k
1
24
MSN Hard
IM
51k
382
16
66.32
Table 4: Dataset statistics comparison
6.2 STATIC SCENE SEGMENTATIONS IN 2D
We evaluate the segmentation quality of EISEN and compare it to other unsupervised segmentation methods: Slot attention (Locatello et al. 2020), unsupervised SAVi (Kipf et al. 202T), and DOM (Bao et al. (022).. To evaluate segmentation quality, compute both the Adjusted Rand Index (ARI) and mean Intersection over Union (mloU) metrics, following previous works in unsupervised 2D segmentations. We evaluate the metrics for both the foreground and background segmentation. As shown in Table 5 and Figure g Slot attention and unsupervised SAVi remain struggling with segmenting the Playroom objects. DOM uses optical flow self-supervise the segmentation masks. However, it underperforms RISEN and outputs incomplete object segmentations, especially for objects with complex geometry details. The output segmentations of the baseline methods are not accurate enough to be used for training downstream object radiance fields.
6.3 STATIC SCENE SEGMENTATIONS IN 3D
We evaluate the quality of learned 3D scene representations using scene segmentations 3D. The segmentations of uORF and MORF obtained by first volume-rendering density maps of objects and background, and then assigning the pixel to the one with highest density. Table 5 shows the ARI and mloU metrics. To reflect the 3D nature, report both metries reconstructed imput views and
15

---

Under review as a conference paper at ICLR 2023
Frame
EISEN segmentation
Ground Irulh Segmentation
Figure 7: Playroom video frames, EISEN segmentations, and ground truth segmentations
Play
Play Plam--vveeeeel
Segments
view
Novel view
Input
Novel view
ARI
f
ARIS
f
1
mloU 1
StotAm
7.7
7.5
22.5
Unsup. SAVi
2.0
14.6
2.0
14.8
DOM
73.6
$7.2
71.9
56.4
EISEN
93.3
87.3
92.2
87.4
MORP (ours)
3D
91.5
84.6
79.1
72.7
90.7
84.6
71.8
66.4
Table Quantitative comparison seene segmentations in 3D
novel views. MORF outputs more accurate segmentations than uORF. EISEN segmentations only compared the input view. We observe EISEN is slightly better than MORF, MORF's 3D segmentations slightly larger than the ground truth due to bilinear sampling in pixel conditioning.
6.4 GENERALIZATION TO REAL IMAGES
We go beyond the synthetic data test the pretrained model's generalization real images. Given few real photos taken using a cellphone, we first use a unsupervised pretrained EISEN to compute the segmentations, followed by novel views reconstruction using pretrained MORF, We show the reconstructions and 3D segmentations Figure T0] Our method able to predict plausible geometry and segmentations for all the objects from a novel view.
6.5 ADDITIONAL EXPERIMENT ON MULTISHAPPNET--AA -ASS
In addition the Playroom dataset, we train and test model the MultiShapeNNt--aas dataset (MSN-Fasy) Stelzner et al. 202T). uORF and pixelNeRF are also trained and evaluated for compari- son. We train the models on 80,000 training scenes and report the novel view synthesis metrics on 500 validation scenes.
We cannot train EISEN the MSN dataset since there is no motion. We thus train EISEN on the COCO dataset and use it directly on the MSN dataset. The segmentations for training are thus flawed; However, even with imperfect segmentation masks, we demonstrate that the reconstruction quality significantly better than uORF and comparable to Pi in Figure
Model
LPIPS 1. SSIMI PSNR
wORF
0.328
0.799
24.8
pixcINeRF
0.229
0.881
29.2
MORE
0.185
0.855
27.5
Table 6: Reconstruction metric
16

---

Under review as a conference paper at ICLR 2023
Input image
SlotAttn
SAVI (uncond.)
DOM
EISEN
GT
Figure 8: Qualitative results on static scene segmentation, in 2D
17

---

Under review as a conference paper at ICLR 2023
Playroom
Image
UORF
MORF
BISEN
GT
Input view
Novel view
N/A
Input view
Novel
N/A
Playroom-novel
Input
view
Novel view
N/A
Input
Noval
view
N/A
Figure 9: Qualitative results on static scene segmentation in 3D
18

---

Under review as a conference paper at ICLR 2023
Input image
Reconstruction (input view)
3D segmentation (input view)
Reconstruction view)
segmentation (novel view)
Figure 10: Novel view synthesis and 3D segmentations on real images taken using cellphone
Input view
Novel view
Input view
Novel view
GT
uORF
PixelNaRF
MORF
Figure 11: Novel view synthesis on MultiShapeNet--aayy
19


Reviews:
[{"review_id": "pt57YkKrJp", "reviewer": "Reviewer_kstW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new framework for 3D object centric learning by leveraging the 2D object mask extracted from the motion information. The proposed approach then trains separate object and background nerf using different network structure. Experiments on data with more complex texture demonstrates the performance of the method.", "summary_of_the_review": "This paper tackles a challenging problem, unsupervised 3D object-centric learning. The proposed framework is convincing. However, it largely depends on the 2D mask information obtained from a 2D image segmentation framework and the object compositional nerf is not new. Thus, the reviewer is concerned that there is not sufficient novelty in the paper.", "strength_and_weaknesses": "#Strength \n+ The paper leverages the motion information for 3D object centric learning. It can handle images with complex textures\n\n+ The object nerf and background nerf are trained based on global object feature and pixel-wise features.\n\n \n#Weakness\n+ Novelty of the paper. The proposed approach highly depends on the 2D segmentation mask which provides significant information for 3D object centric learning. Moreover, the 2D object masks are obtained from an existing work and the formulation for the object compositional nerf is not new [R1], which renders the proposed method lacks novelty. Please refer to the following missing reference for the compositional nerf [R1, R2].\n\n +  [R1] Unsupervised Discovery of Object Radiance Fields, ICLR2022\n + [R2] Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering, ICCV2021.\n\n+ The 2D mask segmentation framework highly depends on the quality of the optical flow. Then it cannot handle the images of pure color.\n\n+ Eq. (4). Should ${\\bf c}$ be ${\\bf c}_i$? \n\n+ It would be great to explain why results from the proposed approach cannot recover sharp boundaries and the reconstructed objects are larger than the real one (see Fig. 4)\n\n+ It is hard to interpret the scene editing results. It looks like all views are different from the ones shown on the top? It makes it difficult for the reader to understand whether the images are rendered from changed objects’ positions or just the changed camera view point.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gCajvwHY05J", "reviewer": "Reviewer_dyX9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends previous work on 3d scene decomposition and reconstruction from a single rgb image. Being able to reconstruct the 3d structure of a scene from a single rgb enables a host of new functionality such as re-rendering multiple views of the scene, editing the scene, and potentially generating images for training 2d and 3d inference models. This work is novel in that it generalizes previous work to arbitrary \"movable\" objects rather than relying on a supervised model trained on a fixed set of objects.  This is a step function improvement in capability enabled by this work. ", "summary_of_the_review": "This is a straightforward combination of two techniques that enables a step-function capability in 3d scene reconstruction from rgb images. I think this is a solid contribution and worth accepting at ICLR. ", "strength_and_weaknesses": "The strengths of this paper are as follows:\n* Allows for scene rendering from a single image and scene editing. \n* Enables a new level of capability by generalizing previous work to handle multiple and arbitrary classes of objects in a scene rather than specific objects trained in a supervised fashion.\n* Convincing quantitative and qualitative results demonstrating that the approach works as expected. \n\n\nThe weaknesses of this approach are as follows:\n* All results are demonstrated only on synthetic data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WHipvDOyr-", "reviewer": "Reviewer_NgLD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Movable Object Radiance Fields, by using the EISEN method to generate object masks and processing object and background separately. Comparing with uORF and PixelNeRF methods on three self-generated datasets, the authors show that the proposed method can extract accurate object geometry and has higher performance in scene reconstruction and editing.", "summary_of_the_review": "The proposed algorithm is not well compared and demonstrated.", "strength_and_weaknesses": "Strength:\nThis paper proposes a Movable Object Radiance Fields, which separate 3D scenes into objects and background before reconstruction. The authors claim that this method can better handle complex and diverse multi-object 3D scenes.\n\n\nWeakness:\n1.The overall structure takes advantage from many advanced algorithms, e.g.. object mask generation, object latent code computation, compositional rendering. I think the authors should better clarify the novelty and unique contributions of their method. \n\n2.The authors carry out their experiments only on self-generated datasets. I think that for better comparison, they should test on more public datasets, e.g. the ShapeNet dataset, the DTU dataset, etc.\n\n3.The authors use a long description to introduce the EISEN segmentation method. I think it would be better if they explain why EISEN, not other methods are chosen to for mask generation in the proposed method. Besides, they should provide the evaluation results for segmentation quality of EISEN; maybe even apply some more segmentation methods to show the influence of object masks (right now they only compare EISEN with GT masks without quantitative mask qualities).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CP0ioHoVfM", "reviewer": "Reviewer_Kin7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new model for unsupervised object-centric 3D scene understanding, called MORF. Unlike previous methods, it leverages 2D image segmentation masks from the pretrained optical flow method EISEN. These are used to initialized a slot based representation augmented by pixel features (as in PixelNeRF), which are used to condition NeRF decoders for objects and backgrounds. On a novel synthetic 3D dataset called Playroom, the model yields better reconstructions and 3D geometry results when compared to prior work. Improved scene editing capabilities are demonstrated, as well.", "summary_of_the_review": "Overall, the paper represents an interesting first attempt at utilizing both video and 3D cues for better unsupervised scene understanding. While I believe its novelty is somewhat overstated, the experiments are competently executed. I am therefore leaning towards acceptance.", "strength_and_weaknesses": "Strengths:\n - The paper addresses a relevant and interesting problem, namely, unsupervised object-centric 3D scene understanding.\n - It is well written and easy to follow.\n - The idea of incorporating motion cues for better segmentations makes sense and has not been explored for 3D models so far.\n - MORF convincingly outperforms uORF on the new dataset.\n\nWeaknesses:\n - The characterization of prior datasets (\"They only demonstrate simplistic scenes with a single object category, and the objects in the scene are in uniform colors.\") is not accurate. While they are synthetic and might indeed by called simplistic, e.g. Multishapenet from ObSuRF and MSN-hard from OSRT use objects from multiple ShapeNet categories, and their standard, multi-colored textures.\n - Consequently, I disagree that the new dataset represents the jump in complexity that the paper suggests it does. To me, it appears of a similar level of complexity as ObSuRF's Multishapenet, and much simpler than OSRT's MSN-hard.\n - Increased performance comes at the cost of requiring video data for training, which the baseline models don't.\n - The fact that the processing of video data is outsourced to an existing 2D model decreases MORF's technical novelty, and doesn't make use of the full potential of combining 3D with video observations.\n - Given the emphasis on novel view synthesis in the evaluation, the state of the art model SRT (Sajjadi et al., 2021) should be included as a baseline, even though it will not yield segmentations or 3D geometry in its default configuration.\n - Some additional information could be added for clarity (see below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject