Unsupervised 3D Scene Representation Learning via Movable Object Inference
Authors: Honglin Chen
Wanhee Lee
Hong-Xing Yu
Rahul Mysore Venkatesh
Joshua B. Tenenbaum
Daniel Bear
Jiajun Wu
Daniel LK Yamins
Venue: Accepted by TMLR ()

Abstract: Unsupervised, category-agnostic, object-centric 3D representation learning for complex scenes remains an open problem in computer vision. While a few recent methods can discover 3D objects from a single image, they remain struggling on scenes with diverse and complex object configurations as they discover objects mostly by appearance similarity which is insufficient for textured objects. In this work, we propose Movable Object Radiance Fields (MORF), aiming at scaling to complex scenes with diverse categories of objects. Inspired by cognitive science studies of object learning in babies, MORF learns 3D object representations via movable object inference. While obtaining 3D movable object signals requires multi-view videos of moving objects, we propose lifting a 2D movable object inference module that can be unsupervisedly pretrained on monocular videos. Thus, MORF requires only multi-view images of static training scenes. During testing, MORF can discover, reconstruct, and move unseen objects from novel categories, all from a single image of novel scenes. We propose a challenging simulated dataset with a diverse set of textured objects for training and testing. Experiments show that MORF extracts accurate object geometry and supports realistic object and scene reconstruction and editing, significantly outperforming the state-of-the-art.

Published in Transactions on Machine Learning Research (03/2024)
Unsupervised 3D Scene Representation Learning via Movable Object Inference
Honglin C Wanhee Lee*l Hong-Xing Yu', Rahul Venkatesh', Joshua B. Tenenbaum",
Daniel M. Bear Jiajun Wu', Daniel L. K. Yamins!
Staanford University
MIT
*Contributed equally
Reviewed OpenReview: http:p///ppaaaaiiis..e.e///mmmmr
Abstract
Unsupervised, category-agnostie, object-centric 3D representation learning for complex scenes remains an open problem in computer vision. While few recent methods can discover 3D objects from a single image, they remain struggling on scenes with diverse and complex object configurations as they discover objects mostly by appearance similarily, which is insufficient for textured objects. In this work, we propose Movable Object Radi- ance Fields (MORF), aiming at scaling to complex scenes with diverse categories of objects. Inspired by cognitive science studies of object learning in babies, MORF learns 3D object representations via movable object inference. While obtaining 3D movable object signals requires multi-view videos of moving objects, propose lifting 2D movable object in- ference module that can be unsupervisedly pretrained on monocular vidcos. Thus, MORF requires only multi-view images of static training scenes. During testing, MORF can dis- cover, reconstruct, and move unseen objects from novel categories, all from a single image of novel scenes. We propose a challenging simulated dataset with a diverse set of textured objects for training and testing. Experiments show that MORF extracts accurate object geometry and supports realistic object and scene reconstruction and editing, significantly outperforming the thate-of-the-art.
1 Introduction
Learning object-centric 3D representations of complex scenes a critical precursor to a wide range of appli- cation domains in vision, robotics, and graphics. The ability to factorize scene into objects provides the flexibility of querying the properties of individual objects, which greatly facilitates downstream tasks such visual reasoning, visual dynamics prediction, manipulation, and scene editing. Furthermore, hypothesize that building factorized representations provides strong inductive bias for composit ional generalization (Gr- eff et al., 2020), which enables understanding novel scenes with unseen objects and configurations.
While supervised learning methods (Ost et al., 2021; Kundu et al., 2022; Müller et al., 2022) have shown promise in learning 3D object representations (such as neural radiance fields (Mildenhall et al., 2020)) from images, they rely on annotations of spocific object and scene categorics. A recent line of work (Yu et al., 2022; Stelzner al., 2021) has explored the problem of unsupervised discovery of object radiance fields. These models can be trained from multi-view RGB RGB-D images to learn object-centric 3D scene decomposition without annotations of object segments and categories. However, these methods remain struggling with textured objects and objects from diverse categories. A fundamental reason is that they heavily rely on self-similarity of visual appearance to discover object entities, which limits their scalability beyond simple texture-less objects with similar shapes.
In this work, we aim to scale unsupervised 3D object-centric representation learning to complex visual scenes with textured objects from diverse categories. To this end, we propose Movable Object Radiance

---

Published in Transactions on Machine Learning Research (03/2024)
Input image
Object radiance fields and geometry (bottom)
Reconstruction
Move object
Replace object
Replace
Top: input view;
Figure 1: Illustration of unsupervised category-agnostic object-centric 3D representation learning. Given single image (an unseen real image here, although our model is trained on synthetic images), we infer object radiance fields that allow photometric and geometric 3D reconstruction. This factorized representation enables 3D scene manipulation, including moving objects, replacing objects, and replacing the background.
Fields (MORF), which learns to infer 3D object radiance fields from single image. Rather than appearance similarity, MORF uses material coherence under everyday physical actions as its underlying object definition, ie., an object is movable as whole in 3D space (Spelke, 1990). However, it is challenging to obtain learning signals to directly infer movable objects 3D, which requires multi-view videos of moving objects. To address this challenge, we propose to lift 2D movable object discovery to 3D. MORF integrates a pretrained unsupervised 2D movable object segmentation module with differentiable neural rendering bridge the gap between 2D segmentation and 3D movable object representations. We show that our lifting design allows learning 3D object-centric representations in complex category-agnostic scenes without multi-view vidcos.
Specifically, MORF integrates recent unsupervised 2D) movable object segmentation method, FISEN (Chen et al., 2022), which is pretrained on optical flow from unlabeled videos. The EISEN module is pretrained to segment objects an image by percoptually grouping parts scene that would move as cohesive wholes. To lift the 2D movable object inference 3D, MORF learns condition 3D radiance field generation with 2D segmentations. We extract object-centric latent representations from segmented images and generate object radiance fields from the factorized latents. To allow high-quality reconstruction of textured objects, our learned latent object representation consists of both entity-level latent and pixel-level latent that better encodes spatially varying appearance details.
To evaluate our method, we propose a challenging simulated dataset Playroom with a diverse set of textured objects, going beyond scenes with simplistic object appearances considered by most current unsupervised 3D object discovery methods (Yu et al., 2022; Stelzner al., 2021; Sajjadi al., 2022b). We demonstrate that MORF can learn high-quality 3D object-centric representations in complex visual scenes, allowing photomet- ric and geometric reconstruction for these scenes from single views. Moreover, our learned representation enable 3D scene manipulation tasks such as moving, rotating, replacing objects, and changing the back- ground. Beyond systematic generalization to unseen spatial layouts and arrangements, we further show that MORF is able to generalize to unseen object categories and even real images while maintaining reasonable reconstruction and geometry estimation quality (Figure 1).
In summary, our contributions are throc-fold. First, we propose scaling unsupervised objoct-centric 3D representation learning to scenes with textured objects from diverse categories by discovering objects with coherent motions. Second, we propose Movable Object Radiance Fields (MORF), which lifts 2D movable object discovery to 3D to remove the requirements of multi-view videos of dynamic training scenes. Third, we demonstrate that MORF allows photometric and geometric reconstruction and editing of complex 3D scenes with textured objects from diverse unseen categories on the Playroom dataset.
2 Related Work
Unsupervised 2D object discovery. Our method is closely related to recent work on unsupervised scene decompasition, which aims to decompose multi-object scenes into separate object-centric representations
2

---

Published in Transactions on Machine Learning Research (03/2024)
Background
NeRF
Training
Inference
Background
Scone NeRF
BG MLP
nownee, otgecte
Object
On ymage
Plel
NaRRF
EISEN
80
Composn
ON 2 wrge
view
MLP
80
reoi.
Image
Sianne
editing
Refersnos image
(a) 2D decomposition
(b) Litting to 3D
(c) Compositiona rendering
Figure Illustration of Movable Object Radiance Fields (MORF). MORF takes as imput single image scene with potentially diverse objects, and infers 3D object and background radiance fields. (a) MORF integrates an image-based movable object inference method, EISEN, which infers a set of object masks. Masked images are used to generate object radiance fields. (b) MORF generates object radiance fields conditioned the latent object and pixel codes. (c) During training, MORF reconstructs the novel view via compositional rendering and is supervised by reconstruction losses. During inference, MORF takes single view of scene, and infers object and background radiance fields in a single forward pass.
from images without human annotations. Most works formulate the problem as learning compositional generative models in the 2D image space. They decompose a visual scene into a set of localized object- centric patches (Eslami al., 2016; Crawford & Pineau, 2019; Kosiorek al., 2018; Lin et al., 2020; Jiang et al., 2019a) 2017; Engelcke et al., 2019; Locatello et al., 2020; Monnicr et al., 2021; Jiang et al., 2019b). The scene mixture models typically generate single-object RGBA images and blend them to reconstruct the full-scene images using iterative inference with recurrent networks (Burgess et al., 2019) or set-based convolutional encoders (Locatello et al., 2020). However, these methods have so far been unable to scale to complex real-world images. A recent branch of work on self-supervised or unsupervised object segmentations explores additional supervision signals such as motions and depth for learning object segmentations (Bear et al., 2020; Kipf al., 2021; Chen et al., 2022; Bao et al., 2022; Elsayed et al., 2022; Ye al., 2022). However, these 2D methods are not aware of the 3D nature of scenes, and thus they do not provide 3D understanding of the underlying scenes.
set of scene mixture components (Burgess al., 2019; Greff al., 2019; 2016;
Unsupervised 3D object discovery. Discovering objects from image collections has been a long-standing topic in computer vision, but previous work on object discovery (a.k.a. co-segmentation) represents objects as 2D image segments without 3D information (Russell et al., 2006: Sivic et al., 2005; 2008; Grauman & Darrell, 2006; Joulin et al., 2010; Rubio al., 2012; Vicente al., 2011; Rubinstein et al., 2013; Cho et al., 2015; Li et al., 2019; Vo et al., 2020). Recently, some attempts have been made to discover 3D object representations. A related set of methods focuses on 3D reconstruction from single image (Ye et al., 2021; Kulkarni et al., 2019; 2020; Kanazawa et al., 2018; Wu et al., 2021). However, it requires strong category- specific shape priors, making it difficult to scale to complex real-world data. Elich et al. (Elich et al., 2020) infer object shapes (Park et al., 2019) from a single image of a scenc. Chen et al. (Chen et al., 2020) extend Generative Query Network (Eslami el al., 2018) to decompose 3D scenes. In particular, the closest. to our work are those that focus on inferring 3D neural object representations from single images (Yu et al., 2022; Smith et al., 2022; Stelzner et al., 2021) sparse views (Sajjadi et al., 2022b). However, these methods rely visual appearance to discover object entities. This fundamental assumption makes it difficult for them to scale to complex scenes with textured objects, diverse object categories, or objects under different lighting, In contrast, our approach leverages motions as the underlying object concept, which is category-agnostic and generalizable to different object appearances.
Scene de-rendering. Aiming to provide full scene understanding, a line of scene de-rendering works have shown reconstructing 3D object-centric representations in specific types of scenes (Wu et al., 2017; Yao
3

---

Published in Transactions on Machine Learning Research (03/2024)
et al., 2018; Kundu et al., 2018; Ost et al., 2021; Kundu et al., 2022: Yang et al., 2021; Wu et al., 2022). Recently, Ost et al. (Ost et al., 2021) propose to represent dynamic scenes scene graph, where each node encodes object-centric information. Muller et al. (Müller et al., 2022) recover 3D object information such as shape, appearance, and pose in autonomous driving scenes. However, these methods rely on mamual annotations of object categories (such as cars) and scene categories (such as street scenes). Similarly, Gkioxari et al. (Gkioxari al., 2022) propose a method that learns to predict the 3D shape and layout of 3D objects relying on the supervision of the 2D bounding box. Our approach only requires object motion in video to infer object segmentations during training, without requiring manual annotations.
Neural scene representations and rendering. Our object representation is based on recent progress in neural scene representations (Park et al., 2019; Mescheder et al., 2019; Sitzmann et al., 2019) and neural rendering (Tewari et al., 2020). Neural scene representations implicitly model 3D scenes using the parameters of deep networks, which could be learned from only 2D images (Niemeyer al., 2020; Sitzmann et al., 2019) with differentiable rendering techniques (Kato et al., 2020; Tewari et al., 2020). Specifically, Neural Radiance Fields (NcRFs) (Mildenhall et al., 2020) have shown photorcalistic scene modeling of static scenes using only images. The most. relevant works in this line aim to infer NeRFs from a single image (Yu et al., 2020; Kasiorek et al., 2021; Rematas et al., 2021). While these works focus on single objects or holistic scenes, we address decomposing a multi-object scene without human supervision. Another relevant line of work aims to incorporate NeRFs into compositional models (Niemeyer & Geiger, 2020; 2021), such as GIRAFFE (Niemeyer Geiger, 2020). While they target scene synthesis, we instead focus on multi-object inference, which GIRAFFE cannot address (Yu et al., 2022).
3 Movable Object Radiance Fields (MORF)
We now describe the problem formulation and our approach, Movable Object Radiance Fields. Given a single input image of a scene that potentially has objects from diverse categories, our goal is to factorize the scene into a set of object-centric radiance fields. To allow unsupervised decomposition of such complex 3D scenes, we propose learning to discover 3D movable objects by integrating and lifting unsupervised 2D movable object inference. The lifting is done by learning to condition 3D radiance fields with 2D object segments using neural rendering.
Movable Object Radiance Fields has three stages. First, it decomposes the input image by inferring its 2D object segmentation masks (Figure 2a). The segmentation extraction network is trained with only optical flow to group scene elements that often move together into an object segment. Second, for each mask, MORF learns an object radiance field with object and pixel latent codes for object features and locally varying details, respectively (Figure 2b). Finally, these object radiance fields are then composed to re-render the scene from multiple views, supervised by the reconstruction loss (Figure 2c). We now describe each component in detail.
3.1 Movable object inference in 2D
Given a single RGB image, we first compute 2D object segmentation mask, represented as a set of H x W dimensional binary-valued tensor {M.) where K is the number of object masks. We denote the background mask as My which can be computed by taking the complement of the object masks union: M, = (U,,,,))
We adopt the EISEN architecture (Chen et al., 2022) to generate high-quality object segmentation masks. The core idea of EISEN is to construct high-dimenssonal plateau map representation (of shape H' W x Q) for each image, in which all feature vectors belonging to the same object are aligned (i.c., have cosine similarity s 1) and all feature vectors belonging to other objects are nearly orthogonal (cosine similarity & 0). Given this representation, the object segments can be easily extracted from a plateau map by finding clusters of vectors pointing in similar directions.
More specifically, EISEN first applies a convolutional backbone to the input image obtain a feature grid, followed by an affinity prediction module that computes pairwise affinities between the features at pixel
4

---

Published in Transactions on Machine Learning Research (03/2024)
location (i.j) and the features at its neighboring locations within a local window. Then EISEN constructs graph, with nodes represented by Q-dimensional feature vectors and edges represented by pairwise affinities. A paassig-passsnng graph neural network is run on the graph to construct the plateau map representations, passing excitatory messages that align the feature vectors of nodes belonging to the same object and inhibitory messages that orthogonalize the feature vectors of nodes belonging to distinct objects. Once the plateau map representation is obtained, EISEN imposes winner-take-all dynamics on the plateau map to extract object segments. We refer the readers to Chen et al. (2022) for more implementation details of EISEN.
During training, EISEN learns pairwise affinities from optical flow estimates from RAFT (Teed & Deng, 2020) network pretrained on Sintel (Mayer et al., 2016). Consider a pair of scene elements (a, b) that project into image coordinates (i,j) and (i',j") respectively. If only one is moving, it is usually the case that they do not belong to the same object; when neither is moving, there is no information about their connectivity. We use this physical logic to construct a paírwise supervision signal for FISEN's affinity matrix. The EISEN training loss is the masked row-wise KL divergence between the predicted and the target connectivity. Although EISEN requires a frame pair as input for computing optical flow during training, it only requires single statie image during inference time for computing the object segmentation masks, and subsequently prodicting the object radiance fields.
3.2 Lifting by learning to condition radiance fields
We represent 3D objects and the background in scene as neural radiance fields (NeRFs). We incorporate and lift. the 2D movable object inference by learning to condition the NeRFs with the 2D segments. To allow high-fidelity reconstruction of complex scenes with spatially varying appearances for objects, each conditioning latent feature consists of both an entity-level slot-based latent and pixel-level local latent, as shown in Figure 2b. In the following, we elaborate on the latent representations.
Object-level latents. Given imput image I and object masks j inferred by a frozen pre-trained FISEN, we extract a set of object latents (where i = 0 denotes the background) { each of which represents object-level features such as shape and position. We denote masked image through M, as L, which is visualized on the left of Figure 2b. We extract an object latent s; by a convolutional backbone encoder E, that produces a feature map f, = EL,),, and average-pool it as S, = avgpool(f.). We further allow the object latent to be adaptively updated by a learnable Gated Recurrent Unit (Chung et al., 2014) (GRU): s; GRU(S; u) for a few iterations, where denotes an update signal given by aggregating features f that are within the object mask:
HxW Σ W W P' = (M Σ M()),
HxW
where V denotes a learnable linear projection matrix. Note that we use shared GRU and a shared V for all objects, while we use separate ones for the background because of the inherent appearance and geometry differences between foreground objects and background.
Pixel-level latents. Object-level latents fall short of capturing spatially varying fine details. Thus, we also learn to extract pixel-level local latents (x) for each object to condition the 3D object radiance field. To obtain the pixel latent codes, we use convolutional encoder F. that produces a feature map g; = F, (L.) For each 3D query point x on a camera ray, we follow Yu et al. (2020) to retrieve the image fcatures of each object by projecting onto the image plane coordinates (x)) and extract latent codes (x from the feature map g, by bilinear interpolation.
Conditioning NeRFs. Since the geometry of the background and foreground objects are inherently differ- ent, we paramcterize the background NoRFs and object NeRFs using two separate conditional MLPs fo and for respectively:
fb (x, d, - (a,c), :x, - (a,c),
(1)
where x denotes a 3D position and d denotes view direction. The MLP parameters of the object NeRFs are shared across all foreground objects.
5

---

Published in Transactions on Machine Learning Research (03/2024)
3.3 Compositional neural rendering
At each point in the rendering of scene, the final pixel value is combination of contributions from each individual scene element to the 3D volumes projecting to that point. We follow uORF (Yu et al., 2022) and take a weighted average of the individual components to obtaín the combined density a and color c. The composite volumetric radiance field is then rendered into a 2D image via the numerical integration of volume rendering by sampling S discrete points along each pixel ray parameterized as r(t) = o o (d, with the ray origin and view direction d. The points on each ray are sampled between pre-specified depth bounds It. ty, with distance 6, between adjacent samples along the ray. Thus, the final pixel color is given by:
c Σ T: exp(- spp--ord]]ec,
Ti =expp= Σ aros);
(2)
Σ Pjoj, c=
E PJoss Pj P
oj -on"" j.k (o...., K}.
(3)
where we let i 0 denote the background quantity, and similarly for j and
3.4 Loss function
During training, we randomly select a single image of a scene as input and render multiple novel views. We train the model using both reconstruction loss and perceptual loss: L , + XpG, with Ap = 0.006. The roconstruction loss the L2 loss between the rendcred image and the ground-truth image L, = =||-- Since reconstruction loss is sensitive to small geometric imperfections and often results in blurry reconstructions, especially due to uncertainties in rendering a novel view, we add a perceptual loss term to mitigate this problem. We compute the perceptual loss as Cp [lex - ex (f) where ex (+)) is k-th layer of an off-the- shelf VGG16 (Simonyan & Zisserman, 2014) image encoder with frozen pre-trained weights.
4 Experiments
We demonstrate our approach three tasks: (a) novel view synthesis, (b) scene geometry reconstruction, and (c) editing scenes by moving objects, replacing objects, and changing the background.
Datasets. We generated four variants of a complex synthetic scene dataset using the ThreeDWorld simula- tion environment (Gan et al., 2020). Each scene includes four camera views with a random azimuth angle and a fixed elevation; the camera always points to the center of the scene. In addition, we also train and evaluate our models on the MultiShapeNeet dataset (Stelzner et al., 2021).
Playroom contains wide range of realistically simulated and rendered objects. Each scene includes 3 objects randomly sampled from set of 2000 object models. These models are drawn from a wide set of categories and have a range of complex 3D shapes and textures. They are placed in random positions and poses in rooms selected randomly from a collection of indoor environments with varying 3D room layouts and floor/wall textures. In each scene, one object is invisibly pushed to generate object motion. There are 15,000 scenes for training and 600 for testing.
Playroom-novel contains mixture of both novel object instances from categories seen during training, well as object instances from novel categories for evaluating the generalization performance of the models. Object instances from novel categories are randomly sampled from 36 held-out categories. We randomly sample 100 novel objects in total. They are placed in the same room environments seen in the Playroom dataset. There are 600 scenes for evaluation.
Playroom-diverss diverse extends Playroom dataset to include more diverse background environments. Playroom dataset uses environments, whereas Playroom-dived diverse contains 30 unique background environments. There are 30,000 additional scenes for training and 600 for testing.

---

Published in Transactions on Machine Learning Research (03/2024)
Playroom
Playroom-novel
Playroom-diverse
MultiShapeNet
Models
LPIPS}
SSIMI
PSNR
LPIPS)
SSIM
PSNR
LPIPS1
SSIM
PSNR1
LPIPS/
SSIM
PSNR
OORF
0.348
0.634
21.5
0.350
0.636
21.6
0.553
0.547
18.1
0.307
0.802
25.1
pixeINeRF
0.250
0.745
24.4
0.265
0.725
23.1
0.357
0.650
20.4
0.159
0.901
30.2
SRI
0.352
0.704
21.3
0.355
0.704
21.3
0.451
0,582
18.7
0.342
0.829
26.1
MORF-nopix
0.244
0.735
23.0
0.264
0.722
22.2
0.466
0.602
18.4
0.317
0.813
25.3
MORF (ours)
0.161
0.784
24.5
0.189
0.755
22.9
0.248
0.675
21.3
0.147
0.881
28.9
Table 1: Novel view synthesis performance measured by LPIPS, SSIM, and PSNR metries on four datasets.
Playroom-edit designed evaluate model's ability to manipulate object radiance fields and synthesize novel images. The dataset contains scenes that result from three types of editing: moving objects, replacing objects, and changing the background. For moving objects, randomly pick one object in the scene and teleport it to random position. For replacing objects, we switch randomly selected object with an object from a different scene. For changing the background, we similarly switch the background with that of another scene. For each scene editing task, we render 200 test scenes for evaluation.
Nell is challenging multi-object dataset in which the objects are shapes from the ShapeNet dataset. Each scene contains 2-4 upright ShapeNet objects sampled from the chair, table, and cabinet classes, for a total of approximately 12,000 objects (Stelzner et al., 2021). We train the models on 80,000 training scenes and report the quantitative results on 500 validation scenes. We cannot train EISEN on the MSN dataset since there is no motion. We thus replace EISEN with Mask2Former (Cheng et al., 2021) model pretrained on the COCO dataset to compute the 2D segmentations of the dataset.
Baselines. We compare MORF to the slot-conditioned unsupervised object radiance field method uORF (Yu et al., 2022) and nonfactorized pixel-conditioned method pixeINeRF (Yu et al., 2020). Both methods learn radiance fields from RGB images without ground-truth object annotations or depth supervision. We also compare to an ablated version of MORF trained without pixel latents to illustrate the benefits of local conditioning in 3D representation learning. We adopt the same training procedures and hyperparameter choices as reported in the original papers. For fair comparison, all models receive input images with the same resolution and are trained with the same batch size.
4.1 Novel view synthesis
We randomly select one camera view of each scene as input and use the remaining three images as ground truth to evaluate the quality of novel view synthesis. In addition to uORF and pixelNeRF, we also compare with the pixel feature ablated version of MORF, All models are evaluated using the standard image quality metrics PSNR, SSIM (Wang et al., 2004), and LPIPS (Zhang et al., 2018).
Results. As shown in Table 1 and Figure MORF outperforms the bascline methods both quantitatively and qualitatively. Figure 3a and Figure 3b show the novel view synthesis results on Playroom and Playroom- novel. uORF and SKT (Sajjadi et al., 2022a) are able to learn rough object decomposition and position estimates, but fail to represent the object shapes accurately, resulting in inaccurate rendering from novel views. The version of MORF without pixel features as conditioning performs second best. Although both uORF and pixel abature-ablated MORF use object latent conditioning, the latter attains substantially better reconstructions, suggesting that accurate object segmentations help constrain the optimization of the neural implicit function. Both MORF and pixeINeRF are able to render novel views reasonably well. We highlight that pixelNeRF's reconstructions are blurry both in background regions (such as floor tiles) and objects (such as the dumbbell), while MORF's reconstructions are sharper. Figure 3c shows the reconstructed novel views on MulliShape:Nel dataset. Both uORF and the pixel feature-ablated MORF fail to decompose the scenes into multiple object slots, resulting in blurry reconstructions the objects. This demonstrates the importance of accurate object dccomposition in high-quality novel view synthesis. The comparison to the pixel feature-ablated MORF illustrates the advantage of using pixel features over object latents in capturing fine-grained details of scenes.
7

---

Published in Transactions on Machine Learning Research (03/2024)
aa)
(b)
(c)
(d)
Input view
uORF
PixeINeRF
SRT
MORF-nopix
MORF
GT novel view
Figure 3: Novel view synthesis results (a) Playmom, (b) Playrom--ooull (c) Playroom-dinerse, and (d) Multi- ShapeNet datasets. The objects Plagroom-novel dataset are not included the training set. MORF outperforms the baseline models both foreground and background reconstruction. MORF is better reconstructing fine- grained textures and object geometries than the other models.
4.2 Geometry reconstruction
To evaluate the quality 3D scene representations, we evaluate both scene segmentations in 3D and extracted meshes from cach model's learned density field. The sogmentations of uORF and MORF are obtained by first volume-rendering density maps of objects and the background, and then assigning the pixel the with the highest density. To extract the meshes of each object, first compute the density field by

---

Published in Transactions on Machine Learning Research (03/2024)
(a)
(b)
(c)
N/A
N/A
Input view
uORF
PixeINeRF
MORF-nopix
MORF
GT
Figure 4: 3D mesh reconstruction results on (a) Playroom, (b) Playroom-novel, and (c) MultiShapeNet datasets. MORF produces more accurate mesh reconstructions than uORF and pixelNeRF, as well as the ablated model of MORF trained without pixel features conditioning. PixeINeRF only outputs single mesh encompassing both the foreground objects and the background. We remove the background mesh in pixeINeRF and visualize the foreground objects only. Mesh reconstruction for SRT is not applicable due to the lack of explicit geometry representation. The ground-truth meshes for MultiShapeNet are unavailable and therefore shown as N/A.
evaluating the foreground decoder grid points in the world coordinate system, followed by the marching cubes algorithm (Lewiner et al., 2003). For fair comparison, all models evaluated using the same grid size of 256.
Metrics. To evaluate the qualily of scene segmentations in 3D, we compute the mean Intersection over Union (mloU) between the segmentations of a novel view and the ground-truth segmentations. In addition, we evaluate the quality of the meshes using Chamfer Distance (CD) (Sun et al., 2018). We compute two types of CDs, object mesh Chanfer Distance (O-CD) and scene mesh Chamfer Distance (S-CD). The O- CD metric measures the quality of mesh reconstruction for each individual foreground object, whereas the

---

Published in Transactions on Machine Learning Research (03/2024)
Left: static segmentations in 2D
Right: static segmentations in 3D
(a)
Input view
Novel view
N/A
(b)
Input
Novel view
N/A
(c)
Input
N/A
view
Input image
2D segments
OORF
MORF-nopix
MORF
GT
Figure 5: 3D static The left shows the 2D scene segmentations of the input view predicted by EISEN (Mask2Former for MultiShapeNett) which are used to conditionally infer 3D object radiance fields. The right shows the scene segmentations for both the input view and the novel view. Note that both uORF and MORF-nophe nopix collapse on MultiShapeNeet and thus they cannot generate meaningful object segmentations.
segmentation results on (a) Plagroom, (b) Playroom-novel, and (c) MultihhppNeet datasets.
S-CD metric measures the quality of the objects' layout in 3D. Due to the lack of object decomposition, pixelNeRF only outputs a single mesh encompassing both the foreground objects and the background. For a fair comparison of S-CD with MORE, we remove the background meshes by setting the density of the grid points below valuuue threshold to zero before applying marching cubes. We search for the threshold with the best S-CD on a validation set, and use the threshold to calculate pixcINeRF's S-CD.
Results. We show the quantitative results in Table 2. MORF outperforms all methods in terms of mloU, O-CD, and S-CD. Figure 4 shows the quality of mesh reconstructions. uORF (Yu et al., 2022) is only able to recover the coarse geometry of the objects. pixeINeRF (Yu et al., 2020) tends to miss the fine details of small objects and thin objects, while MORF can learn more fine-grained object geometry given a single image.
10

---

Published in Transactions on Machine Learning Research (03/2024)
A. Move object
Edited scene (input view)
Edited scene (novel view)
Input image
B. Replace object
Edited scene (input view)
Edited scene (novel view)
Input image
C. Replace background
Edited scene (input view)
Edited scene (novel view)
Input image
uORF
MORF-nopix
MORF
GT
Figure Qualitative results three and novel views are shown in the top and bottom rows respectively. MORF is able to manipulate individual object radiance fields generate novel scenes. Both uORF and MORF (without pixel features) are able perform scene editing to a limited extent, but their reconstructions are blurry. This comparison shows the advantage of using both EISEN segmentations and pixel latents for scene editing tasks.
editing lasks. The syuthesis of the edited from both input
For pixelNeRF (Yu et al., 2020), O-CD is not ported as it lacks object decomposition. Geometry reconstruction metrics are not reported for SRT, since SRT docs not have explicit geometry representations. Figure 5 shows the quality of scene segmentations Compared to MORF, uORF and the pixel feature- ablated version produce less accurate scene segmenta- tions on Playroom and Playroom-nove Furthermore, they both collapse in training and, thus, fail to decompose the scene into multiple objects MultiShapeNel..
Models
Playroom
Playroom-nove
mloUt O-CD 4 S.CD 1 mloUt O-CD 4 S.CD)
HORF pixeINeRF MORF-nopix
50.1
0.324
0.113
58.4
0.324
0.150
0.133
0.128
65.4
0.239
0.096
61.8
0.260
0.122
MORF (ours)
72.7
0.208
0.078
66.4
0.224
0.110
Table 2: Quantitative results on geometry reconstruction
11

---

Published in Transactions on Machine Learning Research (03/2024)
MORF outputs less accurate scene segmentations MultiShapeNet because the 2D segmentations from the pretrained Mask2Former model are prone to generalization error.
4.3 Scene editing
We consider three scene editing tasks: moving objects, replacing objects, and replacing the back- ground. For moving and replac- ing objects, follow the protocol of uORF (Yu et al., 2022) and se- lect the object that has the largest IoU with the ground-truth masks of the target object for editing.
Move object
Replace object
Replace background
Models
LPIPS4.
SSIM PSNR1
LPPPS/ SSTM
PSNR
LPIPS]
SSIM1 PSNR1
dORP
0.381
0.573
20.7
0.384
0.375
20.8
0.371
0.593
20.4
MORF-nopix
0.302
0.706
21.9
0.328
0.701
21.4
0.319
0.700
20.9
MORF (ours)
0.223
0.758
22.7
0.250
0.751
22.1
0.232
0.758
21.7
Table 3: Quantitative comparison on scene editing tasks
pixeINeRF (Yu et al., 2020) not comparable on scene oditing duc to the lack of object dccomposition. We report LPIPS, SSIM, and PSNR on the Playroom-edil dalaset. Ground-truth masks are only used to select which object to edit. FISEN segmentation masks from a pretrained model are used in the feedfor- ward pass of MORF all the images. We show the results in Table 3 and Figure 6. MORF outperforms uORF on all metrics across the three editing tasks. MORF trained without pixel features results in blurry reconstruction on some of the objects.
4.4 Ablation studies
MORF's performance on Playroom drops slightly (ex- cept for O-CD) when object latents are not used. In contrast, pixel latents crucial for MORF to per- form well both novel view synthesis and geometry reconstruction. This implies that although object- centricity provides strong optimization constraint for learning object radiance ficlds, local pixel fcatures are still critical for modeling the fine details of object textures and shapes.
Mask types
slot pixel feat. feat.
Playroom
L,IPS 1 SSIM, PSNR1 O-CD 1 S-CD J
EISEN
/
X
0.266
0.727
22.8
0.239
0.096
EISEN x
0.170
0.776
21.3
0.198
0.079
EISEN
0.162
0.781
24.4
0.208
0.080
GT
0.140
0.791
25.0
0.147
0.036
Table 4: Ablation study results.
Finally, compare MORF trained with EISEN segmentation masks with MORF trained with GT masks. Unsurprisingly, the latter shows better performance in all metries. Qualitatively, we observe that MORF occasionally misses small objects or parts, which is likely direct consequence of EISEN failing to accurately segment these fine-scale scene elements. This indicates that improving unsupervised 2D segmentation should lead to further improvement of MORF's 3D representation learning.
5 Conclusion
In this work, we propose Movable Object Radiance Fields (MORF), which scales unsupervised 3D object- centric scene representation learning to complex and diverse multi-object scenes. Our key technical innovation is allow Icarning 3D representations by integrating and lifting 2D object segmentation module. We show that once high-quality segmentations are obtained through an independent unsupervised module, we can learn 3D object-centric representations in complex category-agnoste scenes. This innovation addresses the limitations of the previous mothods and allows substantial progress to be made on this difficult and important problem.
We demonstrate that MORF enables faithful photometric and geometric reconstruction of scenes with unseen configurations from a single view, generalizes well unseen object categorics, and supports complex editing tasks. We believe that our positive results suggest the promise of further scaling unsupervised 3D factorized representation learning more complex scenes.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Acknowledgments
This work is in part supported by NSF RI #2211258 and #2338203, ONR MURI N00014-22-1-2740, ONR N00014-23-1-2355, the Stanford Institute for Human-Centered Al (HAI), Google, and Samsung.
References
Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discover- ing objects that can move. Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11789-117955 2022.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, et al. Learning physical graph representations from visual scenes. Advances in Newral Information Processing Systems, 33, 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene docomposition and representation. orXiv preprint arXivv:901..0 113990 2019.
Chang Chen, Fci Deng, and Sungjin Ahn. Learning to infer 3d object models from images. arXiv preprint arXiv:2006.0611 06130, 2020.
Honglin Chen, Rahul Venkatesh, Yoni Friedman, Jiajun Wu, Joshua Tenenbaum, Daniel LK Yamins, and Daniel M Bear. Unsupervised segmentation real-world images via spelke object inference. In Proceedings of the European Conference. on Computer Vision (ECCV), 2022.
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv preprint arX6::2112.1076, 2021.
Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In Proceedings of the IEEE conference on computer vision and poltern recognition, pp. 1201 1210, 2015.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555, 2014.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Cathrin Elich, Martin R Oswald, Mare Pollefeys, and Jörg Stückler. Semi-supervised learning of multi-object 3d scene representations. arXiv preprint XXx:20100040300 2020.
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Groff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprinl arXiv:2206.0776, 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmnar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv preprint ar XxXi:19907110055 2019.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukeuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast. scene understanding with generative models. arXiv preprint D.XXv:1603.08577 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scone represcntation and rendering. Science, 2018.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: platform for interactive multi- modal physical simulation. arXiv preprint ar Xiv:2007 Xii:22077.4954, 2020.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Georgia Gkioxari, Nikhila Ravi, and Justin Johnson. Learning 3d object shape and layout without 3d supervision. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognation, pp. 1695 1704, 2022.
Kristen Grauman and Trevor Darrell. Unsupervised learning of categorics from sets partially matching image ures. In 2006 IEEE Compuler Sociely Conference on Computer Vision and Pullern Recogration (CVPR 006), 2006.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Jürgen Schmidhuber, and Harri Valpola. Tagger: Deep unsupervised perceptual grouping, arXiv preprint Kki::660600672, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. arXiv preprinl Xrv: 11:i::708..000 03498, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05900 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. Scalable object-oriented sequential generative models. Unknown Journal, 2019a.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprinl arXie:1910 2210.02384, 2019b.
Armand Joulin, Francis Bach, and Jean Ponce. Discriminative clustering for image co-segmentation, In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognation, 2010.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-3311 1-68,, 2018.
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Difierentiable rendering: survey, Xr preprint arXiv:2006. (ttt:2000.1205555 12057, 2020.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint 2tff::11111220 1rXtc:2111.12504, 2021.
Adam R Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. arXiv preprint Xrxiv:1866.01 0179, 2018.
Adam Kosiorck, Hciko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo J Rezende. Nerf-vae: A geometry aware 3d scene generative model. arXiv preprinl arXiv:2104.0055 2021.
Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2202 2211, 2019.
Nilesh Kulkarni, Abhinav Gupta, David Fouhey, and Shubham Tulsiani. Articulation-aware canonical sur- face mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 452 461, 2020.
Abhijit Kundu, Yin Li, and James M Rehg. 3d-renn: Instanco-level 3d object reconstruction via render-and- compare. In Proceedings of the IEE. conference on computer vision and pattern recogrition, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. arXiv preprint arXio:2205.0 2022.
Thomas Lewiner, Hélio Lopes, António Wilson Vieira, and Gcovan Tavares. Efficient implementation of marching cubes' cases with topological guaranlees. Journal of graphics lools, 8(2):1 15, 2003.
Bo Ti, Zhengxing Sun, Qian Li, Yunjie Wu, and Anqi Hu. Group-wise deep object co-segmentation with co-attention recurrent ncural network. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 8519 8528, 2019.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. orXiv preprint drXiv:2001.02407, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arx:::000 15055, 2020.
Nikolaus Mayer, Eddy llg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dalaset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4040-4048, 2016.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d rectnon in function space. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 1460044770 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint or arXiv:2003.089) 2020.
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry, Unsupervised layered image decomposition into object prototypes. arXiv preprint STf::2104.145755 2021.
Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Autorf: Learning 3d object radiance fields from single view observations. arXiv preprint arXiv:2204.0335 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. arXiv preprint arXiw:2011.12100 2020.
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decompased generative neural radiance fields. arXiv preprint arXiv:2103.11 17269, 2021.
Michael Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Differentiable volumctric ren- dering: Learning implicit. 3d representations without 3d supervision. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3504 3515, 2020.
Julian Ost, Fahim Mannan, Nils Thuercy, Julian Knodt, and Felix Heide. Neural scone graphs for dynamic scenes. Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learn- ing continuous signed distance functions for shape representation. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pp. 165 174, 2019.
Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from single view. Xiv preprint ar arXiv:2102.0888 08860, 2021.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Michael Rubinstein, Armand Joulin, Johannes Kopf, and Ce Liu. Unsupervised joint object discovery and segmentation in internet images. In Proceedings of the IEEE conference on computer vision and pottern recognition, 2013.
Jose Rubio, Joan Serrat, Antonio López, and Nikos Paragios. Unsupervised co-segmentation through region matching. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Bryan C Russell, William T Freeman, Alexci A Efros, Josef Sivic, and Andrew Zisserman. Using multiple segmentations to discover objects and their exlent in image collections. In 2006 IEEE Computer Sociely Conference on Computer Vision and Pattern Recognition (CVPR 06), 2006.
Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR, 2022a. URL https://srt www.ttt.a................ github.io/.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučié, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. arXiv preprint ar Xrx:22206 06922, 2022b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXi:: 1556, 2014.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Welzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems, pp. 1121-1132, 2019.
Josef Sivic, Bryan Russell, Alexei A Efros, Andrew Zisserman, and William Freeman. Discovering objects and their location in images. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, 2005.
Josef Sivic, Bryan C Russell, Andrew Zisserman, William T Freeman, and Alexei A Efros. Unsupervised discovery of visual object class hicrarchies. In 2008 IEFE Conference on Computer Vision and Pattern Recognition, 2008
Cameron Smith, Long-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. orXiv preprint arXiv:2205.0800 089223, 2022.
Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29 56, 1990.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint ar Xi1:21004001148, 2021.
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402 419. Springer, 2020.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. Computer Grophics Forum, 39(2):701-77) 2020.
Sara Vicente, Carsten Rother, and Vladimir Kolmogorov. Object cosegmentation. In CVPR 2011, 2011.
Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In European Conference on Computer Vision, pp. 779-795. Springer, 2020.
16

---

Published in Transactions on Machine Learning Research (03/2024)
Zhou Wang, Alan C Bovik, Hamid R. Sheikh, and Eero P Simoncelli. Image quality assessment: from error
visibility to structural similarity. IEEE transactions image processing, 13(4):600-0) 612, 2004.
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the
IEEE Conference Computer Vision and Pattern Recognation, 2017.
Qianyi Wu, Xian Lin, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Object- compositional neural implicit surfaces. orXiv preprint arXiu:2207.0965 2022.
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. arXiv preprint arXi::2107.18844 2021.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning abject-compositional neural radiance field for editable scene rendering. In Proceedings of the TEEE/CVI International Conference on Compuler Vision, pp. 13779-13788 2021.
Shunyu Yao, Tzu Ming Harry Hsu, Jun-Yar Zhu, Jiajun Wu, Antonio Torralba, William T Freeman, and Joshua Tenenbaum. 3d-aware scene manipulation via inverse graphics. arXiv preprint arXiv:1805 1808.0935, 2018.
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for un- supervised video decomposition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2657-26666 2022.
Yufei Ye, Shubham Tulsiani, and Abhinav Gupla. Shelf-supervised mesh prediction in the wild. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8843 8852, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from few images. arXie preprint arXiv:2012.02) 021900 2020.
Hong-Xina Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
17

---

Published in Transactions on Machine Learning Research (03/2024)
A Appendix
A.1 Dataset details
The Playroom dataset generated with ThreeDWorld (Gan et al., 2020) using custom code. The dataset contains 15,000 training scenes and 600 test. scenes. Each scene includes objects randomly sampled from a set of unique 2,086 object models in 231 object categories. The objects have complex geometry and textures. Table 5 and Figure 7 show the comparison between the Playroom dataset and the MultiShapeNet-- Easy dataset. We also generate additional dataset Playroom-diverse containing 30 unique background environments, 10 times more than the Playroom dataset.
We simulate video frames for each scene, with a randomly selected object invisibly pushed at the first frame to generate object motion. At each time step, we render views at resolution 128 with random camera azimuth angles. is important to note that we only use the video frame pairs for training FISEN. Once EISEN trained, only a single static image is required for inference. We only use static images at single time step for learning radiance fields. Figure 8 shows examples of the video frame pairs from the dataset, along with the EISEN segmentations and ground truth segmentation masks.
Playroom
Playroom-diverse
MultiShapeNet- Easy
Figure 7: Qualitative comparison between Playroom and MultiShapeNet.
environment maps
object per
Datasel
scenes
objects
categories
views
scene
Playroom
15,000
2,000
231
3
3
4
Playroom-diverse
30,000
2,000
231
30
3
4
MultiShapeNeet-ass
80,000
12,000
3
2-4
3
Table Dataset statistics comparison.
A.2 Implementation details
Convolutional backbone encoder. To compute object and pixel latents, we use two separate convo- lutional encoders to extracts feature pyramids from the input image respectively. We show the encoder
18

---

Published in Transactions on Machine Learning Research (03/2024)
Frame
Frame 2
EISEN segments
segments
Figure 8: Playroom video frames, EISEN segmentation, and ground truth segmentation.
architectures in Table 6 and Table 7. All the encoder weights from initialized from scratch without any pretraining. All the activation layers are ReLU.
NeRF decoder. We adapt the NeRF architecture from uORF (Yu et al., 2022). We show the architecture detail in Figure 9. The input to the NeRF network consists of positional embeddings, ray directions, object latents, and pixel latents. In all experiments, we set the positional embedding frequency to 5. Note that do not apply high-frequency encoding to the ray directions. The feature dimension of object and pixel latents are set to 64 respectively. Following uORF, the background decoder is slightly different from the foregorund decoders in that it does not have the second last layer and third last layer.
Layer name
Input shape
Output shape
Concatenate
convi
64x64x7
64x64x64
conv2
64x61x61
32x32x61
conv3
32x32x61
16x16x64
conv4
16x16x64
16x16x64
Upsample
16x16x64
32x32x64
conv5
32x32x128
32x32x61
with conv2
Upsample
32x32x64
64x64x64
conv6
64x64x128
64x64x64
with convi
Table Backbone encoder for computing object latents.
A.3 Experiment details
Training. In each batch, we input one image and render 2 reconstructed images for supervision, one from the input view and the other from a randomly selected novel view. We render each pixel with 128 samples. For perceptual loss, we uses the output of the fourth convolutional block in a VGG16 (Simonyan & Zisserman, 2014) pretrained ImageNet. We Adam optimizer with initial learning rate 0.0003. We adopt
19

---

Published in Transactions on Machine Learning Research (03/2024)
Layer name
Input shape
Output shape
Concatenate
blocki
64x64x3
32x32x64
block2
32x32x64
16x16x64
block3
16x16x61
8x8x128
block4
8x8x128
4x4x256
Upsample
4x4x256
8x8x256
with block3
Upsample
8x8x384
16x16x384
with block3
Upsample
16x16x448
32x32x448
with block2
Upaample
32x32x448
64x64x448
with block1
Linear
64x64x512
64x64x64
Table Backbone encoder for computing pixel latents.
(x)
30
256
+
256
256 -
256 -
256 - 256 +
256 - 256 ..>
RGB
64
$
64
Figure NeRF decoder network archilecture. Input shown in blue, output vectors are shown in red. The imput vector a concatenation of positional embeddings 7(x), ray direction d, pixel latents p, and object latents The number inside each block indicates the dimension of the vectors. Black arrows indicate layers with ReLU activations, orange arrows indicate layers with no activation, dashed black arrows indicate layers with sigmoid activation. aI" indicates vector concatenation.
shown in green, full-connetted hidden layers are
linear learning rate warm-up for the first 1000 iterations. The learning is exponentially decreased by half for every 200 thousand iterations. We train our model with batch size of 16 for 500 thousand steps, which takes about 10 days 8 NVIDIA A40 GPUs.
Inference Figure 10 shows the inference procedure. MORF takes a single view of a new scene as input and infers object and background radiance fields in a single forward pass. Given an arbitrary camera pose, a novel view can be synthesized via compositional rendering.
Spatial sampling for memory efficiency. Rendering multiple radiance fields for all the objects in is memory expensive, especially when there are large number of objects per scene. To improve the memory efficiency of the training, we only compute object radiance field at small subset of world coordinates. Specifically, we only compute object radiance field at world coordinates whose projected 2D coordinates onto the input view resides in the segmentation mask the corresponding object. Otherwise, we set the density as zero. This significantly reduces for the number of forward passes through the NeRF decoder networks. We add padding around the object segments to account for potential incomplete segmentation due to occlusion.
Metric details. We use PSNR and SSIM from the Pytorch Image Quality (PIQ) package as in uORF. LPIPS is computed using the code provided by the LPIPS authors Zhang el al. (2018) after normalizing the pixel values to [-1, 1]. We compute Scene Chamfer Distance (S-CD) on the foreground object meshes in a scene. Points are uniformly sampled on the mesh surface create dense point cloud; then N points are randomly sampled from the point cloud, where N is 1,024 per object mesh and 3,072 per scene mesh. We normalize the point cloud coordinates into unit cube before CD calculation.
20

---

Published in Transactions on Machine Learning Research (03/2024)
Background
NeRF
Backgrund
Pist
BG MLP
Object
Obj maae
Phul
NeRFs
EISEN
o
Composo
Otjy Image"
00
FG MLP
00
ONj. image
0g
view
00
2D decomposition
(b) Lifting to 3D
Compositional rendering
Figure 10: Model inference. MORF takes as imput a single image of a scene with potentially diverse objects, and infers 3D object and background radiance fields. (a) MORF infers set of object masks using pre-trained segmenter.
Masked images are used to generate object radiance fields. (b) MORF generates object radiance fields conditioned the latent object and pixel codes. (c) MORF reconstructs the novel view via compositional rendering.
Playroom
Playroom-novel
Models
ARI
mIoU
ARI
mIoU t
SlotAttn
7.7
22.8
22.5
Unsup. SAVi
2.0
14.6
2.0
14.8
DOM
73.6
57.2
71.9
EISEN
93.3
87.3
92.2
87.4
Table 8: Quantitative comparison of scene segmentations in 2D.
A.4 Static scene segmentation in 2D
We evaluate the segmentation quality of EISEN and compare it to other unsupervised segmentation methods: Slot attention (Locatello al., 2020), unsupervised SAVi (Kipf al., 2021), and DOM (Bao et al., 2022). To evaluate segmentation quality, we compute both the Adjusted Rand Index (ART) and mean Intersection over Union (mloU) metrics, following previous works in unsupervised 2D segmentations. As shown in Table 8 and Figure 11, Slot attention and unsupervised SAVi remain struggling with segmenting the Playroom objects. DOM uses optical flow to self-supervise the segmentation masks. However, it underperforms EISEN and outputs incomplete object segmentations, especially for objects with complex geometry details. The output segmentation of the baseline methods are not accurate enough to be used for training downstream object radiance fields.
A.5 Additional qualitative results
More novel view synthesis results Figure 12 shows additional view synthesis results on Playroom dataset. Figure 13 shows additional results on Playroom-diverss dataset. Figure 14 shows additional results MultiShopeNNet dataset. For each image, the top shows the reconstructions from the input view, the bottle row shows the reconstructions from a randomly selected novel view.
Scenes with different numbers of objects We create additional testing scenes with random number of objects between and We show MORF able to generalize well test scenes with a different number of objects. Figure 15 shows qualitative results of the challenging scenarios with 4 to 6 objects in scene.
21

---

Published in Transactions on Machine Learning Research (03/2024)
Input image
SlotAttn
SAVI (uncond.)
DOM
EISEN
GT
Figure 11: Qualitative results on static scene segmentation 2D.
Scenes with object occlusion The training and validation sets contain examples with partial object occlusion since we are rotating the camera with random azimuth angles. We show qualitative examples of partial object occlusion in the first two rows of Figure 16. To further test the ability of MORF handling scenes with more object occlusion, we create additional test scenes with objects placed closer to each other, as shown in the last two rows of Figure 16. MORF is generally capable of handing partial object occlusion and performing approximate amodal completion of the occluded objects. The quality of geometric reconstruction slightly deteriorates compared to the unoccluded objects in the input view.
Scenes with object stacking Playroom dataset simulates an infant playing environment with toys placed on the floor. In real-world scenarios, objects are often stacked on lop of one another. We create additional scenes with three types of stacking configuration: (a) a cup on top of coaster, (b) a food item on the plate, and (c) an object on top microwave. We generate 1000 scenes for each stacking configuration and fine-tune a MORF model on the additional scenes. Figure 17 shows the qualitative results of novel view synthesis on held-out scenes.
22

---

Published in Transactions on Machine Learning Research (03/2024)
Models
LPIPS!,
SSIMT
PSNR1
uORF
0.336
0.577
19.8
MORF-nopix
0.323
0.580
19.9
MORF (ours)
0.064
0.903
31.0
Table 9: Quantitative comparison on real-world scenes
Generalization real world scenes We go beyond the synthetic data to test the pretrained model's generalization real images. Given few real photos taken using cellphone, first use an unsupervis- edly pretrained EISEN to compute the segmentations, followed by novel views reconstruction using MORF pretrained on Playroom. We show the reconstructions and 3D segmentations in Figure 18. Our method is able to predict plausible geometry and segmentations for all the objects from a novel view.
Training on real-world scenes Besides training on synthetic data, we train on a real-world dataset including scenes with four plant pots or vases on tabletop decorated with tablecloths. We gathered 9 plant pots, vases, and 6 tablecloths and captured 745 scenes for training and 140 for evaluation, cach with 3 images from different poses. We cannot. train EISEN this dataset since there is no motion. We thus replace EISEN with Mask2Former (Cheng et al., 2021) model pretrained on the COCO dataset to compute the 2D segmentations of the dataset. Table 9 and Figure 19 show novel view synthesis results on the lesting real-world scenes. MORF synthesizes the novel view accurately, whereas uORF fails to decompose the scenes, resulting inaccurate 3D object radiance fields.
23

---

Published in Transactions on Machine Learning Research (03/2024)
Huus
Input view
uORF
PixeINeRF
SRT
MORF-nopix
MORF
Novel view (GT)
Figure 12: Additional novel view synthesis results on Plagroom
24

---

Published in Transactions on Machine Learning Research (03/2024)
R
Input
uORF
PixelNeRF
SRT
MORF-nopix
MORF
Novel view (GT)
Figure 13: Additional novel view synthesis results on Plagroom-diverr
25

---

Published in Transactions on Machine Learning Research (03/2024)
Input view
uORF
PixeINeRF
SRT
MORF-nopix
MORF
Novel view (GT)
Figure 14: Additional novel view synthesis results on Plagroom
26

---

Published in Transactions on Machine Learning Research (03/2024)
Input (QT)
Inpul (reconslructions)
Novel (reconstructios)
(GT)
Figure 15: Novel view synthesis results of testing scenes with arbitrary objects
Input view (GT)
Input view (reconstructions)
Novel view (GT)
Novel view (reconstructions)
Figure 16: Novel view synthesis results of scenes with object occlusion
27

---

Published in Transactions on Machine Learning Research (03/2024)
Input view (GT)
Input view (reconstructions)
Novel view (GT)
Novel view (reconstructions)
Figure 17: Novel view synthesis results of scenes with stacking configuration
Input image
Reconstruction (input view)
3D segmentation (input view)
Reconstruction (novel view)
3D segmentation (novel view)
Figure 18: Novel view syuthesis and 3D scene segmentattons real images taken using cellphone.
28

---

Published in Transactions on Machine Learning Research (03/2024)
Input view
uORF
MORF-nopix
MORF
Novel view (GT)
Figure 19: Novel view synthesis results on wval-world scenes
29


Reviews:
[{"review_id": "c2iIefn4k9", "reviewer": "Reviewer_hAH5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nThe problem of 3D scene decomposition from a single image is a challenging problem. Instead of using full 3D supervision for metric reconstruction and semantics (the way for example one would use Scannet), the authors use only supervision for 2D semantics and then use the self-supervision of NeRFs to extract one radiance field per object. The authors evaluate on novel view synthesis, mesh reconstruction, and scene editing.\n\nWEAKNESSES\n\nThe main weakness of the paper is the lack of clarity in the method and what exactly happens during training and during inference. The fundamental challenge on 3D scene decomposition is that segmentation and reconstruction is a chicken and egg problem: If one knows the segmentation, then reconstruction is feasible (what the paper here follows) but, on the other hand, segmentation should depend on \nthe geometrically consistent reconstruction. Unfortunately, the current scheme does not allow the segmentation to be informed by the reconstruction. This would require a recurrent unit taking advantage of reconstruction/rendering inconsistencies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about how to extract object-level radiance fields so that objects can be later removed, inserted, and re-arranged in a scene.\nThey propose Movable Object Radiance Fields. The method consists of object segmentation through optical flow and radiance field extraction conditioned on object segmentation. A novel dataset is proposed.\nThe paper belongs to the field of unsupervised scene decomposition. \n\nFor 2D object segmentation, the paper uses EISEN which learns similarities using the RAFT optical flow method. EISEN uses pairs during training but only single images during inference. Then, given predicted object masks and an RGB image,  global and local features are computed that are fed as conditioning variables to two radiance fields, one for the background and one for each object in the foreground.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "0. Respond to the main weakness above.\n\n1. The abstract and the introduction are incomprehensible. It is unclear which problem the authors address regarding input and output during inference and training.\n\n2. \"While supervised learning methods have shown promise in learning 3D object representations (such as neural radiance fields) from images, they rely on annotations\" \n\nThis is really confusing. NeRFs do not use any annotation and the only supervision signal is the views themselves. \n\n3. Fig.2: Please provide an additional figure about what exactly happens during inference.\n\n4. Authors should clarify the dimensions of the feature map and over which dimension pooling is happening (3.2) to obtain $s_i$ from $f_i$.\n\n5. It is not clear what is the meaning of the GRU. An ablation would be illuminating. \n\n5. During the method description it is not clear whether multiple views are used for the NeRF optimization. \n\n6. During inference it is not clear, how an object radiance field is subtracted from or added to the scene.\n\n7. It was not clear whether EISEN is frozen during training. \n\n8. There is no quantitative evaluation in real images.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AC3AlQ3oQe", "reviewer": "Reviewer_qD6L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- MORF is a well designed composition of existing modules that enables state of the art performance. In order to obtain a per-object 3D representation, it makes sense to segment a 2D image into a set of segments based on a learned prior of whether they appear to be movable or not (the EISEN method), and then extracting global and pixel-level features for the segmented objects to condition neural fields.\n- High performance relative to prior methods uORF, PixelNerf, SRT, for scene-level representations.\n\n**Weaknesses**\n\nWhen training and evaluating computer vision systems in the synthetic domain, it's essential to carefully design the data generating process to faithfully represent the key challenges of a real-world perception task. In this paper, the synthetic data is generated in a way that doesn't capture all of these challenges sufficiently, which in turn limits what we can conclude about the utility of this representation for downstream tasks (e.g. manipulation) and the limitations of the proposed approach. \n\nFor compositional 3D representations of scenes of objects, the main challenges are:\n- In terms of encoding semantic properties\n     - generalization to novel environments (different surfaces the objects are placed on) and lighting\n     - generalization to novel object instances of categories seen during training\n     - generalization to instances of novel object categories not seen during training\n- In terms of encoding geometric properties\n     - handling an arbitrary number of objects in a scene\n     - handling camera pose variability\n     - accurately capturing object scales, locations and rotations in 3D - in the real world, objects can have complex spatial relationships like being close and far from each other, as well as being put inside or on top of other objects.\n\nThe current data generation setup is either limited w.r.t. a majority of these challenges, or the details to determine this are missing:\n\n- in Playroom there are only 3 environments and they're used both for training and testing. MultiShapeNet has one environment.\n- in Playroom, there are 100 novel objects, but it's not clear whether these are from novel categories or not, or how different they are from the objects used in the training set. In MultiShapeNet it's not clear whether the training/validation objects are distinct.\n- there are a most 4 objects in each scene\n- the objects appear to be well-spaced, and the camera is at a fixed height. As a result, it appears that object-object occlusion doesn't happen very often, which is unrealistic. \n- objects have similar scales and are placed in canonical poses. For example in the real world, an apple is much smaller than a table or a bowl and is usually found on top of a table or inside a bowl.\n- the camera is at a fixed height, greatly simplifying the scene-level geometric relationships that need to be learned.\n\n\nBuilding up to a synthetic training and evaluation setup that faithfully captures all of these will be a gradual process, but given the nature of the task and the capabilities of current 3D graphics systems and freely available 3D assets, I believe the proposed data generation system should be more complex and diverse.\n\nMinor cosmetic comments and typos.\n- table 1 overflows to margins\n- table 2 and 3 overlap\n- 3.2 paragraph 1 - \"each conditioning latent encode\", it's unclear what this means. Might want to say feature instead of encode\n- 3.2 paragraph 3 - \"convolution encoder\" -> convolutional encoder", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a new method, MORF, for learning compositional 3D representations of scenes of objects. The proposed approach has the ability to render novel views of scenes, extract the 3D shape of objects, move objects, replace objects, and replace the background. The proposed approach consists of \n- unsupervised 2D segmentation (based on the EISEN method, which learns to segment scene entities that appear to be movable)\n- global and local latent feature extraction for foreground objects and background\n- foreground and background neural fields conditioned on local and global features\n- weighted average-based combination of the foreground and background neural field outputs for scene-level rendering\n\n\nIn the first evaluation setting, training is done on synthetic scenes of 3D objects from the Playroom dataset (2000 object instances, 231 categories) in 3 different environments, and evaluation is done on scenes from 100 held out objects. In the second evaluation setting, the model is trained and tested on scenes of 3D objects from MultiShapeNet (12K objects, single simple environment). The proposed approach outperforms prior work at view synthesis quality, 3D object shape reconstruction accuracy, and scene editing tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main requests for changes are: \n- (Critical) Additional explanation in the draft about what is the reasoning behind the design decisions of how the training/testing data is generated w.r.t the points discussed above? How do these decisions limit what we can conclude about the capability of MORF? This might be better than prior work, but what are the arguments justifying the way the task is implemented in the synthetic domain? Is this complex enough, and if so, why?\n- (Between critical and \"simply strengthening the work\", especially because it would require new data and experiments) One of the strengths of synthetic data is that it allows us to gradually make testing data more challenging across different axes (e.g. increasing the number of objects, increasing the amount of occlusion, using completely novel environments). Adding additional results that take advantage of this would greatly improve the paper, and help us understand how well MORF actually works, and what should be the goals of future work.\n\nIf there improvements are added and the results are good, the evidence would be much stronger for the claims in the paper. I am currently selecting that \"Claims and evidence are not supported by accurate, convincing and clear\" evidence, because while the accuracy and clarity is good, there is large room for improvement (in terms of complexity and realism) of the implementation of the task in the synthetic domain (see weaknesses).", "broader_impact_concerns": "Currently, there are no concerns for ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DNKwSS1RWr", "reviewer": "Reviewer_BY36", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The paper is well-written, and I think they are relatively easy to follow. I think reproducibility should be ok as well\n- The algorithm is able to provide wider ranges of tasks, such as generalizing to more classes of objects, etc. Also, experimental results are slightly sharper and higher quality compared with other approaches.\n- The claims in this paper are relatively grounded, and the qualitative results demonstrated that the proposed algorithm is doing well.\n\n[Weaknesses]\n- The paper writing is generally OK, but I don't fully get the motivation of using an unsupervised optical flow approach (EISEN). I know it's one of the state-of-the-art algorithms for such tasks, but utilizing more training data (pairs of images) may grant MORF an unfair advantage over other approaches. It's a little bit challenging to justify the novelties of the approach.\n- The EISEN algorithm is pre-trained using Sintel synthetic datasets, therefore I wonder if the generalization ability of the approach could potentially be better if we pre-train on real-world optical flow datasets.\n\n- nit: What's authors intuition of using multiple FG MLPs for each objects? Will that further reduce the artifacts in the results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed a new unsupervised object-centric 3D scene representation learning framework called MORF. Leveraging 2D unsupervised segmentation approaches, the algorithm disentangles the foreground/background images, and learns different representations for different object instances. The approach is able to provide a wide varieties of the tasks, such as 3D reconstruction, image editing, etc. Experimental results on 3D datasets suggest that MORF has potentials of reconstructing higher quality 3D geometries and have improved scene editing capabilities against other algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There seems to be some poorly formatted documents/texts starting from Sec. 4.3, it's a good idea to fix it for future revisions. The following sections of the paper looks like to be completed in a rush.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After revision the reviews were borderline and the AE read the paper and the reviews to understand the issues. On the one hand, the reviewers thought that the problem was timely, worth tackling and thought that the method was interesting. On the other hand, there were residual concerns about novelty of the contributions and the real-worldness of the results. These concerns were shared by multiple reviewers.

Given the mixed recommendations, the AE has considered the identified weaknesses of the current paper. The AC thinks that the paper's strengths outweigh its weaknesses.
1. *Novelty* TMLR, as a journal, has a strong focus on well-supported claims as opposed to novelty. While the AE does understand the origin of the reviewer concerns about novelty, novelty (in the CVPR/NeurIPS/ICML-sense) by itself carries less strength for TMLR decisions as compared to whether the claims are demonstrated and the method is of interest.
2. *Real-world results* The AE understands the concern about real world experiments -- some things transfer well from rendered scenes and others do not. However, the AE thinks that the additional real-world results have demonstrated the claims sufficiently well for the purposes of this paper and its claims (although improving the realness of the modeling may be important for tackling more complex scenes with photometric interaction as hAH5 points out).

At the same time, the AE does think a small modification would be helpful and would alleviate the concerns of the reviewers about the real world data. The paper would be improved if the final claims-oriented paragraph of the introduction and potentially the abstract were more specific. For instance,  "complex 3D scenes with textured objects from diverse unseen categories" in the introduction would be improved by specifying the datasets on which this is demonstrated. The AE doesn't want to force the authors to write this and it is up to the authors, but the AE would encourage the authors to consider this sort of modification (and is giving "minor revisions" to enable this).
Decision: Accept (poster)