Unsupervised Discovery of Object Radiance Fields
Authors: Hong-Xing Yu
Leonidas Guibas
Jiajun Wu
Venue: ICLR 2022 Poster (2022)
Keywords: object discovery
scene decomposition
3D scene representations
object-centric learning

Abstract: We study the problem of inferring an object-centric scene representation from a single image, aiming to derive a representation that explains the image formation process, captures the scene's 3D nature, and is learned without supervision. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating the complex 3D-to-2D image formation process into powerful inference schemes like deep networks. In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and rendering with deep inference networks for unsupervised 3D scene decomposition. Trained on multi-view RGB images without annotations, uORF learns to decompose complex scenes with diverse, textured background from a single image. We show that uORF enables novel tasks, such as scene segmentation and editing in 3D, and it performs well on these tasks and on novel view synthesis on three datasets.

Published as conference paper at ICLR 2022
UNSUPERVISED DISCOVERY OF OВJЕCT RADIANCE FIELDS
Hlong-Xing Yu Stanford University
Leonidas J. Guibas Stanford University
Jiajun Wa Stanford University
ABSTRACT
We study the problem of inferring object-centric scene representation from single image, aiming to derive a representation that is learned without supervision, explains the image formation process, and captures the scene's 3D nature. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating powerful unsupervised inference schemes like deep networks with the complex 3D-to-2D image formation pro- cess. this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and ren- dering with deep inference networks for unsupervised 3D scene decomposition. Trained on only multi-view RGB images, uORF learns decompose complex scenes with diverse, textured background from a single image. We show that uORF enables novel tasks, such as scene segmentation and editing in 3D, and it performs well on these tasks and on novel view synthesis on three datasets'.
INTRODUCTION
Building factorized, object-centric scene representations is a fundamental ability in human vision and a constant topic of interest in computer vision and machine learning. We identify that such representations should bear three characteristics: they should be learned without supervision or prior knowledge about object categories, and therefore applicable to environments where object categories unknown; they should explain the image formation process, addressing questions like 'what if the object is not there?'; they should be 3D-aware, capturing geometric and physical object properties for navigation, interaction, and manipulation.
For decades, researchers have attempted to solve the problems from various angles. Inspiring as they are, these methods each lack in one more the three aspects (Table 1). Computer vision research on unsupervised object discovery has achieved great success on deriving object segments from real images, but doesn't 't capture the image formation process, nor 3D-aware (Rubinstein al., 2013; Zhu et al., 2012). Recent work on deep probabilistic inference for visual scene decomposition unsupervised and generative (Burgess et al., 2019; Engelcke et al., 2019; Locatello et al., 2020), though most still formulate the problem as 2D segmentation and work on simple scenes of geometric primitives, ignoring the complex 3D nature of realistic visual scenes. A few recent papers on 'scene de-rendering' have attempted to reconstruct 3D, object-centric representations by leveraging the forward rendering procedure (Yao et al., 2018; Ost et al., 2021); they are however supervised, relying on annotations of specific object and scene categories, such as cars and road scenes.
The fundamental challenge that prevents these systems from acquiring all three desired properties is that the image formation process from 3D to 2D is complex and non-differentiable (e.g.. due to occlusion). Thus, for a long time, it has been unclear how it may be integrated with powerful deep inference schemes. But most recently, progresses in neural rendering (Tewari al., 2020) have demonstrated that their continuous, implicit representation works well with gradient-basse based inference models, such as deep networks. In particular, Neural Radiance Fields (NeRFs) (Mildenhall al., 2020) recover 3D scene from a set of RGB images via differentiable volume rendering. Such encouraging advances in generative modeling suggest promising route for inferring 3D, generative, and object-centric scene representations without supervision.
In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating conditional NeRFs as 3D object representations with deep inference networks for unsupervised 3D
"Code and data can be found https: httpa:// /t/p///eyn..c//uuuae/

---

Published as conference paper at ICLR 2022
Input image
Object radiance fields
Reconstruction
Object removal Object insertion
Rearrangement
Top: objects; bottom: background
Top: input view; bottom: novel view
Figure 1: Illustration of unsupervised discovery of Object Radiance Fields. We aim to infer factorized object and background radiance fields from single view, allowing reconstructing and editing of the scene.
scene decomposition. uORF infers a set of object radiance fields and a background radiance field; thus, uORF represents 3D scene composition of object radiance fields (Figure 1). During training, such radiance fields are neurally rendered in multiple views, with reconstruction losses in pixel space training supervision; during testing, uORF infers the set of object radiance fields from a single image. Learning uORF does not require explicit supervision of 3D geometry or object segmentation, but only sparse multi-view RGB images of training scenes.
The integration of NeRFs allows us to work with more realistic scenes with complex object shapes and diverse background environments, beyond simple scenes such as those in multi-dSprites (Greff et al., 2019) and CLEVR (Johnson et al., 2017), as considered by most current unsupervised scene decomposition methods. We further make two innovations to improve uORF's performance. First, as background geometry and appearance can be quite different from foreground objects in 3D, we design uORF with explicit modeling of both components. This background -waree design not only facilitates learning on complex scenes, but also allows single-imag scene editing including moving individual objects and changing background. Second, as volume rendering requires massive queries to render a single pixel for the recomposed scene, practical challenge of learning uORF lies in the computational inefficiency. We tackle this issue by proposing a novel progressive coarse-to tos--toofiin fine training which improves representation quality while remaining affordable computational cost.
We evaluate uORF on factorized scene representation learning (c.g., segmentation in 3D) and scene generation (e.g., novel view synthesis, scene editing in 3D). Our evaluation is three datasets with gradually increasing complexity: first, CLEVR-like scenes with primitives foreground shapes; second, room scenes with complex chair shapes and textured backgrounds; third, more diverse room scenes with various foreground shapes and backgrounds. Our results show that uORF learns factorized representations that can segment 3D scenes into objects with fine shape details (e.g., thin chair legs) and backgrounds with well-recovered appearance details (e.g., irregular textures of wooden floor).
In summary, our contributions are three-fold. First, we propose the problem of inferring an unsu- pervised, factorized, generative, and 3D-aware scene representation from a single image. Second, we introduce unsupervised discovery of Object Radiance Fields (uORF) that infers individual 3D object radiance fields from single view for the proposed problem. Third, we demonstrate that uORF enables novel tasks such as scene segmentation and editing in 3D, and we show that it generalizes to novel scene arrangement and unseen combinations of object properties.
2 RELATED WORK
Co-segmentation and object discovery. Our work is closely related to traditional computer vision methods on object discovery, which aims to locate (visually similar) objects in collection of images. These methods typi- cally model objects as visual words and adopted methods from topic modeling to localize objects (Russell et al., 2006; Sivic et al., 2005; 2008), or cluster and group image patches (Grauman & Darrell, 2006; Joulin et al., 2010; Rubio et al., 2012; Vicente et al., 201 1: Rubinstein et al., 2013; Cho et al., 2015). Recent works have integrated the clustering- based strategy with deep learning (Li et al., 2019; Vo et al., 2020). Nevertheless, they do not explain image formation process nor are they 3D-awrr..
Appreach
Unsup. Gen, 3D
Co-segmentation Deep prob. infer. Scene "de-render"
x
x
Ours
Table 1: Comparison existing methods.
2

---

Published as conference paper at ICLR 2022
1. Object-centric fatent inference
Object radiance fields
I11. Re-composing and re-rendering
Trpout imsge
Backg- latent
Backy
Training on multiple scenes
Compose
Loss
Objoct
Re-rendered
Rederence
latents
Object
Inference
Single forward pass
New sammo
Segment
Novel view
Fdltt
Figure Overview. Our model learns to infer set of latents in single forward pass. II. Each ob- ject/background radiance field consists of latent and shared conditional NeRF. III. During training, recompose the scene and re-render images for supervision. We train our model on different scenes, At test time, we use single image of an unseen scene for reconstruction editing.
Deep probabilistic inference for scene decomposition. Our method is also closely related to deep probabilistic inference scene decomposition. Most works formulate the problem as compositional generative modeling, where visual scene is represented by set of latent codes that either correspond to localized object-centric patches (Eslami et al., 2016; Crawford & Pineau, 2019; Kosiorek et al., 2018; Lin et al., 2020; Jiang et al., 2019) scene mixture components (Burgess al., 2019; Greff et al., 2019; 2016; 2017; Engelcke et al., 2019). Recently, Locatello et al. (2020) proposed the Slot Attention module to simplify the inference by slot-based encoder. Besides these inference models, Monnier et al. (2021) formulated scene decomposition as layered image decomposition and demonstrated on real images. However, these methods do not account for the 3D nature of scenes.
A few methods have recently been proposed for unsupervised 3D scene decomposition. Elich et al. (2020) infer object shapes from single scene image, but they require pretraining on groundtruth shapes. Chen et al. (2020) extend Generative Query Network (Eslami et al., 2018) to decompose 3D scenes, but they require multi-view images during inference. The closest our work concurrent work by Stelzner et al. (2021) which also utilizes slot alot-based encoder and NeRFs as 3D representations. However, Stelzner et al. (2021) relies on groundtruth multi-view dense depth in addition to images in training. Moreover, we explicitly model the separation of objects and background to address various complex shapes and textured backgrounds, while they only demonstrate scenes with single textureless background.
Scene de-rendering. A few recent works have shown reconstructing 3D object-centric representa- tions by incorporating forward image rendering process (Wu et al., 2017; Yao et al., 2018; Kundu et al., 2018: 8; Ost et al., 2021). Yao et al. (2018) de-render an image into semantic segments and geometric object attributes, which enable 3I) scene manipulation. Most recently, Ost et al. (2021) propose Neural Scene Graph represent dynamic scenes into scene graph where each node encodes object-centric information. However, these methods rely on manual annotations of specific objects (such cars) and scene categories (such as street scenes).
Neural scene representations and rendering. Our method is related to recent progresses neural continuous scene representations (Sitzmann et al., 2019) and neural rendering (Tewari et al., 2020). Neural scene representations parameterize 3D scenes with a deep network (Sitzmann al., 2019). Combined with differentiable neural rendering techniques (Kato et al., 2020; Tewari et al., 2020), they can be learned from only 2D images (Niemeyer al., 2020). In particular, Neural Radiance Fields (NeRFs) (Mildenhall et al.. 2020) have shown impressive novel view synthesis. Related follow-up works include those that infer NeRFs from a single image (Yu et al., 2020; Kosiorek et al., 2021; Jang & Agapito, 2021) and those that incorporate NeRFs into generative models (Schwarz et al., 2020; Niemeyer & Geiger, 2021; Chan et al., 2020). Different from these works which cope with single objects or holistic scenes, leam object NeRFs via decomposing multi-object scene without segmentation annotations. GIRAFFE (Niemeyer& & Geiger, 2020) generates object NeRFs and thus compose 3D scenes an adversarial framework. However, targets unconditional generation and cannot tackle inference (see Appendix E), while we focus on single-image inference of multi-object scenes. Thus, we address fundamentally different problem compared GIRAFFE.
APPROACH
Our goal is to infer from a single image a set of object-centric 3D representations to generate the underlying 3D scene. We show illustration in Figure 2. Our object representation is conditional
3

---

Published as conference paper at ICLR 2022
object radiance field. Thus, learn to infer object-centric latents from single image (Figure 2- D) The inferred latents are used to condition a network to yield the 3D object and background radiance fields (Figure 2-1I), forming In training, we compose all object and background radiance fields and render the recomposed scene from multiple views. We obtain supervision by comparing rendered images reference images (Figure 2-I1II without needing 3D geometry or segmentation annotations. We describe each of our model components in the following and leave implementation details in Appendix B.
3D-aware, generative and factorized scene representation.
3.1 OBJECT-CENTRIE LATENT INFERENCE
Our goal is infer latent object-centric repre- sentations from a single input image. We as- sume that an underlying 3D scene is composed of a background environment and no more than K foreground objects. Thus, the output of our object-centrie latent inference process is a latent z" for background and set of latents {z, Ji=1 for foreground objects (empty objects are al- lowed). To encourage unsupervised object-wise factorization, we adopt a slot-base clot-based formula- tion (Locatello et al., 2020). The assumption in this formulation is that objects should share common prior latent space. The main idea include three steps. The first step is to sample all object latents (i.e., slots) from the same prior distribution (background is special object) to encourage representational uniformity across all slots ("sampling"), "). Then each slot is bound to an object region via an attention module ("binding"), In the last step each slot gets updated by the bound object features to specialize for that object ("updating"), Locatello al. (2020) have demonstrated success on segmenting 2D images.
Learnable
1t GRU
N
Sumple
slot
Updaaic
Feature
GRU'
CNN
mlot/
Update
Sample
l'imess
Updato
slot
Atlention weights
Inpui image
Figure 3: Our object-centric latent inference. The atten- tion binds each object's features slot.
However, in 3D scenes, the geometry and appearance of the background are highly different from those foreground objects. Modeling them indistinguishably often leads to object representations entangled with blurry background segments (Burgess al., 2019; Locatello et al., 2020), which impedes applications such as scene editing and re-composition. Thus, we propose a background- aware slot attention module (Figure 3) that separately models objects and environment better capture the compositional structure of 3D scenes.
Background-awarr slot attention for sampling and binding. In the sampling step, we model the latent prior distribution of foreground objects by Gaussian with learnable mean and variance, i.e., we sample slots! ~ N (nl diag(o () E IR R for objects. For latent prior of backgrounds, we learn another Gaussian and sample single slot from i..., slot² ~N" diag(o")) E E I
To bind slots to image features, we let all the slots to compete for explaining the input image representation. To do this, flatten the convolutional feature map (we include details about convolutional encoder Appendix B.1) into a set of N input feature vectors, feat E NxD The slot competition is modeled by key-query attention (Bahdanau et al., 2014):
exp(Mr.s) attn. Σ, exp(
(sloth) T g (slots!)
where M = k(feat)
E RRNx(4+1) (I)
Here l and gd are learnable linear mappings RD-D for computing dot-product similarity (Luong al., 2015), and vD is fixed softmax temperature (Vaswani al., 2017). One as a soft K-means, where attn, softly assigns feature to the slots (centroids). The background slot expected
this process
capture the modality of background features and bind all of them, allowing foreground slots to focus only on the objects without explaining background segments (Figure 3). Besides the representation design, we further encourage disentanglement between background and (during
foregrounds by two additional designs: (1) We represent and query foreground/backgronn the neural rendering process) in different coordinate frames. (2) To discourage object slots from fitting background, we impose a locality constraint in early training. We set a foreground box and enforce that every foreground query point outside the box has zero density. We include details in Appendix B.2.
4

---

Published as conference paper at ICLR 2022
Updating slots to infer latents. With the attention weights, we form the update signal by ag- gregating input values via a weighted mean pooling updatees := WhT efeeet E Rixh where Wh == / attnı,1), and updates := W o (feat) E RKXD, where W == attns,++ attni,j+)). Slots are then updated using the update signals via learnable rule parameterized by Gated Recurrent Unit (GRU) (Cho et al., 2014), that slots + GRU (slots' updates/) and slot" + GRU updates"). We repeat the at- tention computation and updating for iterations, and output all the slots as the final latents z and {z. N=1-
We show pseudo-code of our background awaree slot attention in Appendix (Alg. 1).
3.2 COMPOSITIONAL NEURAL RENDERING
We represent a 3D object as conditional neural radiance field. A NeRF is a continuous mapping g: (x, d) - (c, o) from spatial location x and viewing direction d to emitted color с and volume density used for volume rendering (Max, 1995). This mapping is parameterized by MLP network. We adopt a conditional NeRF g(x, d|z) for our inference scheme (detailed in Appendix B.3). The MLP parameters are shared across all objects gf (x, dlz;), ), but not the background g'(x, dlz²) due to its distinct geometry and appearance distribution.
To compose individual objects and background into the holistic scene, we consider a scene mixture model and use density-weighee mean to combine all components: ā Er W; C Σi WyC where w; o Eij J. Here and č are the combined density and color, respectively, The color C(r) of a camera ray r(t) = o o (t) is then estimated via numerical integration of volume rendering, using& S discrete combined samples along a ray (Max, 1995): C(r) = Σ [i - exp(-o,6:)]c where T; = exp (- Σ k,ō,) Here ō, is the distance between adjacent samples along ray.
K
3.3 MODEL LEARNING
Loss functions. As shown in Figure 2, during training we input a single image of a scene, infer object and background radiance fields, render multiple views from the recomposed scene, and compare them to reference images for loss computation. We train our model across multiple scenes. Our training loss function comprises of reconstruction loss, a perceptual loss, and an adversarial loss: L Lrecon + Apercept percept Autl Lanve where À are weights. The reconstruction loss is frecom = | 1112 where I and I denote the reference image and rendered image, respectively.
Since we estimate 3D radiance fields from a single view, there can be uncertainties about the appearance from other views (e.g., the back view). For example, regarding visual appearance of objects, inaccurate global lighting estimation leads to uncertainties in brightness and shadows from occluded views even if the object shapes can be well estimated. To address this, incorporate perceptual loss (Johnson et al., 2016) which is tolerant to mild appearance changes. The perceptual loss is defined by Ppeceet = P(T) p( [1 where is deep feature extractor (See Appendix B.4).
In addition to appearance, there can be even higher uncertainties in estimating object shapes from a single view, which is multi- modal distribution. In this case, the unimodal reconstruction loss leads to blurry results ("mean shape"). We mitigate this issue by adding an adversarial loss which can deal with mulli- distributions:
Lad- = (()( +
where f(() - log(1 + exp(-()). (2)
Here we adopt the RI regularization (Mescheder et al., 2018) to stabilize training. D denotes a discriminator to distinguish rendered images and reference images I. We iterate between training the discriminator by minimizing Latv and training our inference model (Figure 2) by minimizing L.
Coarse-to-fine Progressive Training. A practical challenge in training compositional NeRFs lies in the computational cost of neural volume rendering, as requires massive queries to render single pixel. While there have been attempts fast inference (Liu et al., 2020; Rebain et al., 2020; Neff et al., 2021; Garbin et al., 2021; Reiser et al., 2021; Yu et al., 202 I), high space complexity in training remains a challenge. Further, because our perceptual and adversarial losses depend image patches, the system has to render a large enough patch (instead of a single pixel) at the same time, which further increases its space demand.
To allow training on a higher resolution, we propose a coarse-to- Searse-to-fine progressive training. In a coarse training stage, we bilinearly downsample reference images low resolution (e.g., 64 64), and train
5

---

Published as conference paper at ICLR 2022
uORF on these downsampled images. Although the coarsely trained model can already decompose the 3D scenes and recover rough object radiance fields, fine details (c.g., thin legs of chairs) might be missing. Thus, in following fine training stage, replace the low-resolution reference images with image patches randomly cropped from high-resolution images (Figure 2-III), and render the correspondingly located patches from recomposed scene radiance fields to compute the loss. We include further training details in Appendix B.4.
EXPERIMENTS
We evaluate uORF both scene representation (via scene segmentation in 3D) and scene generation (via novel view synthesis and scene editing) on three datasets.
Data. We build three synthetic datasets with gradually increasing complexity. For each scene in the dataset, point the camera to the scene center and render four images with a randomly chosen azimuth angle and a fixed elevation angle. We describe more details in Appendix C.I.
CLEVR-567. The first dataset includes scenes of 5-7 CLEVR objects (Johnson et al., 2017), with random position and orientation on clean background. Foreground object shapes include three geometric primitives (i.e., cubes, spheres and cylinders). Since there is intrinsic ambiguity in estimating specularity from a single image, we use only the largely diffuse "Rubber" material. There 1,000 scenes for training and 500 for testing.
Room-Chair, The second dataset includes scenes of 3 to 4 chairs of the same shape in a room with three different textured backgrounds. There are 1,000 scenes for training and 500 for testing.
Room-Diverse, The third dataset includes scenes of diverse foreground object shapes and background appearances. Each scene includes 4 different chairs, whose shape is randomly sampled from 1 1,200 ShapeNet chair shapes (Chang et al., 2015), and the background is sampled from 50 floor textures from the web. There are 5,000 scenes for training and 500 for testing.
4.1 SCENE SEGMENTATION IN 3D
We first evaluate sORF's factorized 3D scene representations via scene segmentation in 3D.
Baselines. Because there is no previous work focusing on the same setting as uORF, we compare to a 2D state-of-the-art scene decomposition model Slot Attention (Locatello et al., 2020) for unsupervised scene segmentation wherever possible (detailed in Appendix C.2). In addition, we compare two ablated versions of uORF, First, remove our muckground-awan modeling but keep the same number of slots. Second, we ablate our progressive training such that the training procedure only contains the coarse training stage. We refer to ablated models as "uORF (w/o background)" and "uORF (w/o prog. train.)", respectively.
Metrics. We adopt the widely-used Adjusted Rand Index (ARI) as our metric. To evaluate scene segmentation in 3D, we consider three kinds of ARIs: (1) For direct comparison to 2D methods, we compute ART on reconstructed images. (2) To reflect the 3D nature, we also compute ARI on synthesized novel views, denoted as "NV-ARI". Note that each scene includes 4 views, and only one is used as input, and the other three are treated as novel views for this metric. (3) In line with previous 2D methods, also report foreground ARI (Fg-ARI), computed only on foreground regions indicated by groundtruth masks. Yet, we note that Fg-ARI cannot fully reflect the segmentation quality, because background segmentation is completely ignored.
density map d' for each slot i. The segmentation label for each pixel 8, is given by Sp = arg max K+1 dj We show results on Table 2 and Figure (more in Appendix D). For all segmentation metrics, we show mean and standard deviation for three runs. uORF outperforms all methods in terms of ARI and NV-ARI. From Figure 4, it is clear that uORF is able to discover the 3D objects from a single image. These results validate that uORF can learn well-factone cell-factorized 3D object-centrie scene representations. Also notice that uORF yields better ARI even in input views compared to 2D slot attention. This is likely due to our background-awar design, as our ablated model "uORF w/o background" has shown similar input-view results compared to slot attention (e.g.. see 3rd and 4th columns in Figure 4).
Results. We volume-
4.2 NOVEL VIEW SYNTHESIS
We then show that uORF is 3D- awaae and generative via evaluation novel view synthesis.

---

Published as conference paper at ICLR 2022
Input
view
Novel
N/A
view
Input
view
Novel view
N/A
Ground truth Color image
Ground truth segmentation
Slot Attention
w/o background w/o prog. train. uORF (ours)
Figure Examples on scene segmentation 3D. Novel view images are for reference but input.
CLEVRR-67
Room-Chair
Room-Diverse
Models
3D metric
2D metric
3D metric
2D metric
3D metric
2D metric
NV.ARIT
ARI
Fy-ARI
NV-ARI
ARI1
Fg-ARI
NV.ARI
ARI
Fg-ARIT
Slot Attention
NIA
3,5:0.7
93.2::..5
N/
38.41:1.4
40,2::455
N/A
17.4:11.3
43.8:211.7
wORF (wfo background)
10.5.13.
11.71..6
86.4128
404192
42.3.10.66
93.3119
21018.11
24.01999
7891131
wORF (wfo prog. train.)
81.1
83.7=088
84.22.0.5
62.3-2.5
65.4226
81.0233
$38114
63.711.77
66.914.1
uORF (ours)
83.8:t0.3
86.3=0.1
87,420.8
74.3=119
78.8:2666
8888:2.7
56.910.2
65.6110
67.91177
Table Scene segmentation results. "NV-ARI" refers ARI evaluated on novel views. "Fg-ARI" refers ARI
evaluated with only foreground pixels. Slot Attention (Locatello al., 2020) is of-te-of-the-ar art 2D method.
Setup. For each test scene, we randomly pick one image as input and the remaining three images as groundtruth for novel view synthesis. As Slot Attention purely in 2D and does not support novel view synthesis, we compare to conditional NeRF (Mildenhall et al., 2020), equipped with a convolutional encoder similar to uORF, termed as "NeRF-AE" (see Appendix C.2). For fair comparison, increase the latent dimension for NeRF-AE to guarantee approximately the same computational cost, and we use the same training strategy and losses as uORF, Thus, NeRF-AE can also be seen monolithic alternative model to uORF. We also compare with the ablated models, "uORF (w/o background)" and "uORF (w/o prog. train.)". We use the perceptual metric LPIPS (Zhang al., 2018), together with SSIM (Wang et al., 2004) and PSNR,
evaluation metrics.
Quantitative results in Table 3 and qualitative results in Figure 5 (more in Appendix D). Quantitatively, uORF outperforms all compared methods on all metrics. From the qualitative comparison in Figure 5, we highlight three advantages of uORF, First, compared with NeRF-AE, which has a monolithic latent structure for the entire scene, uORF better preserves the features of each object: for example, see how NeRF-AE fuses object colors in the first two rows, while uORF does not. This shows the advantage of factorized scene representations to structurally describe a visual scene. Second, compared with uORF (w/o background), one can clearly see how our background-awan modeling helps recovering background appearances: uORF can accurately recover background appearance of the Room-Chair example, while uORF (w/o background) does not. also facilitates learning complex scenes with diverse, textured background: uORF can learn to roughly recover object shapes in the boom-Diverse example. Third, compared with uORF (w/o prog. train.), highlight that the fine training image patches indeed improves both visual quality and representation quality: the full uORF tries to recover sharp edges cubes, while uORF (w/o prog. train.) cannot distinguish cube from sphere.
Results.
7

---

Published as conference paper at ICLR 2022
Input
view
Novel view
Input
view
Novel view
Ground truth NeRF-AE w/o background
w/o prog. train.
vORF (ours)
Figure 5: Qualitative results on scene dccomposition and novel view synthesis. is reconstruction and the second is novel view.
Within every two rows, the first
CLEVR-567
Room-Chair
Room- Dive--
Models
LPIPS
SSIM
PSNR1
LPIPS}
SSIM
PSNR1
LPIPS
SSIM
PSNR
NeRF-AE
0.1288
0.8658
27.16
0.1166
0.8265
28.13
0.2458
0.6688
24.80
uORF (w/o background)
0.0919
0.8924
28.93
0.1671
0.7852
27.86
0.2231
0.6924
25.90
uORF (w/o prog. train.)
0.1044
0.8894
28.84
0.1573
0.8287
28.33
0.2123
0.6760
25.19
uORF (ours)
0.0859
0.8971
29.28
0.0821
0.8722
29.60
0.1729
0.7094
25.96
Table 3: Comparison on novel view synthesis from single image.
Overall, the novel view synthesis results suggest that uORF can Iearn to represent 3D scenes with reasonable fidelity, even with the presence of complex foreground object shapes, such as chairs and different textured backgrounds.
4.3 SCENE DESIGN AND EDITING IN 3D
Being object-centric and 3D-aware, uORF is able to edit 3D scene radiance fields inferred from single view, and generate novel scene images.
Setup. We test sORF's ability to edit scenes and synthesize novel images on the Room-Chair dataset. We consider both moving foreground objects and changing background appearance. For object moving, we randomly pick one object in a test scene and move to a random position. We render 4 images for each of the 500 test scenes. For background changing, background texture to a different one and also render 4 images for evaluation. To indicate the new background, re-pick and re-put foreground objects such that the resultant background indicator image is different from the groundtruth image.
replace the current
For uORF and Slot Attention (Locatello et al., 2020), we use groundtruth masks of the input view only for case of evaluation. We determine which slot to move by picking the one with largest mask IoU. For NeRF-AE (Mildenhall et al., 2020) to do editing. we back project the masks to frustums to determine the 3D regions to be moved/replaced. We use LPIPS, SSIM, and PSNR as our metrics.
Results. We show results in Table
Moving objects
Changing background
Models
4 and Figure 6 (more in Appendix D). Again, uORF outperforms all pared methods on all metrics. As Fig- ure depicts, images synthesized by uORF show least artifacts and highest quality and fidelity.
I.PPSS
SSIM
PSNR1
I.PISS
SSIM
PSNR1
NeRF AE
0.2451
0.7284
23.18
0.2185
0.7132
25.42
Slot Attention
0.3941
0.7134
23.06
0.3689
0.7283
23.94
uORF backgroun))
0.2206
0.7448
0.7719
26.68
uORF (w/o prog. train.)
0.1583
0,8313
28.19
0.1586
0.8306
28.27
uORF (ours)
0.0855
0.8711
29.26
0.0822
0.8729
29.53
4.4 GENERALIZATION AND ANALYSIS
Table 4: Comparison on scene editing.
Finally we explore the generalization ability of uORF. We consider generalization on unseen, chal- lenging spatial arrangement of objects, well generalization on unseen object appearances.
8

---

Published as conference paper at ICLR 2022
1. Move objects
Input view
Novel
N/A
Input image
view
Ground truth
Slot Attention
NeRF-AE
w/o background prog- train.
OORF (ours)
Change background
Input view
Novel
N/A
Background image (imput image is the sume as above)
view
Ground truth
Attention
NeRF-AE
background wio prog. train. uORF (ours)
Figure 6: Qualitative results on single-image 3D scene manipulation. The first two rows are for moving object and the second two rows are for changing background.
Loss functions
ARIT
LPIPS.
Models
ART1
LPIPSS
Models
NV.ARI
ART
59.1=20.5
0.3610
Slot Attention
5.7:1:033
N/A
Slot Attention
N/A
2.2:1:0.6
Rev, PercepL
65.2:10.8
0.2156
(nurs)
85.0.10.3
87.4:10.4
Adv.
OORF (ours)
83.2::0.6
0.1540
PORF (oracle)
85.5
87,5:00.3
Rec. Percept Adv.
65.6:11.0
0.1729
Table 5: Generalization novel
Table Generalization to unseen combinations of color and shape.
Table Ablation study for losses on the Room-Diverse dataset.
challenging spatial arrangements.
Generalizing to challenging spatial arrangements. We build new test dataset, packed-CLEVE 11, where each scene has 11 objects that are closely packed into a cluster. Therefore, each scene bears an unseen number of objects in an unseen challenging arrangement. We models trained on CLEVR-567, report results in Table 5 and Appendix Figure 19. Despite uORF never sees such object arrangements, still achieves a reasonable performance and outperforms baselines.
Generalizing to new combination of shape and color. For unseen object appearances, we con- sider generalization in systematic way such that the model can deal with unseen combination of object color and shape. Thus, we build new training set similar to CLEVR-567, but we remove red cylinders and blue spheres from the object candidate pool. Then dataset with only red cylinders and blue spheres in the candidate pool. We show results in Table and examples in Appendix Figure 20. We see that although uORF has never seen any of the test set objects, it achieves similar results to the one trained on a normal CLEVR-567 dataset (denoted as "uORF (oracle)"). This suggests uORF's ability for systematic generalization to unseen combinations of object color and shape. We further validate generalization unseen object shapes in Appendix D.
trained models on another
Evaluating loss functions. uORF uses perceptual and adversarial losses combat intrinsic uncer- tainties in single-image inference of 3D representations. We show ablation results novel view synthesis in Table 7 and Appendix Figure 18. Both losses significantly improve image quality.
CONCLUSION
In this work, propose unsupervised discovery of Object Radiance Fields (uORF), which learns to infer object-centric 3D radiance fields from a single image of complex multi-object scenes. We demonstrate uORF's ability suggest promising direction to integrate neural rendering into deep probabilistic inference scheme, allowing learning factorized 3DD object-centric scene representations from only RGB images.
segmentation and scene generation in 3D. Our positive results

---

Published as conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was part supported by Qualcomm Innovation Fellowship (QIF), Stanford Institute for Human-Centeres AI (HAI), Stanford Center for Integrated Facility Engineering (CIFE), Toyota Research Institute, a Vannevar Bush faculty fellowship, Amazon, Autodesk, Google, and Bosch.
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our work, have provided the training and test code repository's together with all three synthetic datasets, and pre-trained models on all three datasets. We have also provided detailed instruction using code well training on new datasets. In Appendix B, we describe details for re-implementing our work.
ETHICS STATEMENT
Learning object-centric scene representations is long-standing topic in vision and it finds various applications downstream tasks. We represent a 3D scene as a composition of simple radiance fields, which only models object appearances and entangles their physical properties that may be crucial to downstream tasks in non-interpretable way. However, we envision that careful designs in more structured 3D object representations for specific downstream applications could help improve transparency and human interpretability in model prediction and behavior, allowing both better performances and secure. fair usage. In our code release, we will explicitly specify allowable uses of our system with appropriate licenses. We will use techniques such as watermarking to identify and label visual contents generated by our system.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv: arXiv:1409.044 1409.0473, 2014. 4
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerehner. Monet: Unsupervised scene decomposition and representation. 1901. /1390, 2019. 1,3,4 4
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-gam: Periodic implicit generative adversarial networks for 3d-aware image synthesis. arXiv:2012.00926. 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: an information-rich 3d model reposi- tory. arXiv:
2015. 6, 16
Chang Chen, Fei Deng, and Sungjin Ahn. Learning to infer object models from images. 4rXiv:2006.00130. 2020. 3
Kyunghyun Cho. Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares. Holger Schwenk. and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine tramslation. arXiv: 1078, 2014. 5
Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In CVPR, 2015. 2
Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI, 2019.
Yihe Dong, Jean- Saan-Baptiste Cordonnier. Andreas Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. arXiv:2103.034000 2021. 18
Cathrin Elich, Martin R Oswald, Marc Pollefeys, and Jörg Stückler. Semi-supervised learning of multi-object 3d
scene representations. arXiv:2010.04030 2020. 3
Martin Engelcke, Adam Kosiorek, Oiwi Parker Jones, Ingmar Posner, Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv: 13052, 2019. 1,3
SM Eslami, Nicolas Heess. Theophane Weber, Yuval Tassa, David Szepesvari. Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast seene understanding with generative models, arXiv: /00::6003 0605.0555. 2016. 3
"httpa://gtt /ttp://///ttttttt wtt.......o/oooe////uu
10

---

Published as conference paper at ICLR 2022
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Sci., 2018. 3
Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High- fidelity neural rendering 200fps. arXiv:2 10380, 2021.
Kristen Grauman and Trevor Darrell. Unsupervised learning of categories from sets of partially matching image features. In CVPR, 2006. 2
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao. Jürgen Schmidhuber, and Harri Valpola. Tagger:
Deep unsupervised perceptual grouping. arXiv:1600.0672: 0672440060066, 2016. 3
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber.
Neural expectation maximization.
1788: 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. ICML, 2019. 2,
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. 0rXiv:2/09.01700, 2021. 3
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. Scalable object-oriented sequential generative models. urXiv:1910.02388 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution, In ECCV. 2016.
Justin Johnson. Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: a diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 2, 6, 15
Armand Joulin, Francis Bach, and Jean Ponce. Discriminative clustering for image co-segmentation. In CVPR,
2010. 2
Tero Karras, Samuli Laine, Miika Aittlla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In CVPR, 2020.
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. arXiv:2006. /2057, 2020. 3
Adam R Kosiorek. Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. urXiv::8066 01794, 2018. 3
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Moreno, Rosalia Schneider. Soña Mokrá, and Danilo
Rezende. Nerf-vae: geometry aware 3d scene generative model. arXiv:2104.00587, 2021.
Abhijit Kundu, Yin Li, and James M Rehg. 3d-remn: Instance-level 3d object reconstruction via render-and- compare. In CVPR, 2018.
Bo Li. Zhengxing Sun, Qian Li, Yunjie Wu, and Angi Hu. Group-wise deep object co-segmentation with co-attention recurrent neural network. In CVPR, 2019. 2
Zhixuan Lin. Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition, arXiv:2001.0244 002007,. 2020.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In
NewrIPS, 2020. 5
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv:2006. httl::2006./00555 15055, 2020. 1, 4, 6, 7, 8, 15, 16
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches aatention-based neural machine translation. 1508 04025, 2015. 4
Nelson Max. Optical models for direct volume rendering. JEEE TVCG, 1995.
Lars Mescheder, Andreas Geiger, and Sehastian Nowozin. Which training methods for gans do actually converge? In ICML, 2018. 5
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. 089934 2020. 1,3, 7, 8, 17
11

---

Published as conference paper at ICLR 2022
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry. Unsupervised layered image decomposition into object prototypes. arXiv:2104, 1451:22,44.4555, 2021. 3
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Chakravarty R. Alla Chaitanya, Anton Ka- planyan, and Markus Steinberger. Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks. arXiv:2103.05231, 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. W/Xi:2011.122100 2020. 3, 13, 18
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance fields.
arXiv:2103.17269 2021.
Michael Niemeyer. Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020. 3
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In CVPR, 2021. 1,3
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. DeRF: Decom- posed radiance fields. arXiv:2011.12490, 2020.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. arXiv:2103.13744 2021.
Michael Ruhinstein, Armand Joulin, Johannes Kopf, and Ce Liu. Unsupervised joint object discovery and segmentation in internet images. In CVPR, 2013. 1,2
Jose C Rubio, Joan Serrat, Antonio López, and Nikos Paragios. Unsupervised co-segmentation through region matching. In CVPR, 2012.
Bryan C Russell, William T Freeman, Alexei A Efros, Josef Sivic, and Andrew Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In CVPR, 2006. 2
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. arXiv:2007.022442 2020. 3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
/409. 2014. 14
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: 3ddsttructure-wware neural scene representations. In NewrIPS, 2019.
Continuous
Josef Sivic, Bryan C Russell, Alexei A Efros, Andrew Zisserman, and William T Freeman. Discovering objects and their location in images. In ICCV, 2005. 2
Josef Sivic, Bryan C Russell, Andrew Zisserman, William T Freeman, and Alexci A Efros. Unsupervised discovery of visual object class hierarchies. In CVPR, 2008. 2
Karl Stelzner, Kristian Kersting, Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv:2104.01 2021. 3
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox, What do single-view 3d reconstruction networks learn? In CVPR, 2019. 13
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann. Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, State of the art on neural rendering. CGF, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv:/ 2017.
Sara Vicente, Carsten Rother, and Vladimir Kolmogorov. Object cosegmentation. In CVPR, 2011.
Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In ECCV, 2020. 2
Zhou Wang, Alan Bovik, Hamid Sheikh, and Eero P Simoncelli. Image quality assessment: From error visibility structural similarity. IEEE TIP, 2004.7 7
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In CVPR, 2017. 3
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing- Xue, Bolei Zhou, and Ming- Hssan Yang. Gan inversion: survey. 100iv:2101 00100778 2021. 19
12

---

Published as conference paper at ICLR 2022
Shunyu Yao, Tzu Ming Harry Hsu, Yan Zhu, Jiajun Wu, Antonio Torralaa, William T Freeman, and Joshua Tenenbaum. 3d-aware scene manipulation via inverse graphics. Stti::/888.09333 2018. 1,3
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. Pixelnerf: Neural radiance fields from one or few
images, arXiv:2012.021900 2020. 3, 13
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrces for real-time rendering of neural radiance fields. urXiv:2103. 14024, 2021. 5
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 7
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li, On the continuity of rotation representations neural networks. In CVPR, 2019. 19
Jun Yan Zhu, Jiajun Wu, Yichen Wei, Eric Chang, and Zhuowen Tu. Unsupervised object class discovery via saliency- guidde multiple class learning. In CVPR, 2012. 1
A SUPPLEMENTARY MATERIAL OVERVIEW
In the following supplementary document, we first provide implementation details unsupervised discovery of Object Radiance Fields in Section B. We then describe details on datasets and baseline architectures in Section C. We show additional results in in Section D, including results on general- ization to unseen object shapes, a demonstration on real photos, an analysis on the sensitivity to slot initialization, and additional qualitative results all experiments of the main paper and failure cases. In Section E, we show comparison to GIRAFFE (Niemeyer & Geiger, 2020) to demonstrate that focus on fundamentally different problem (unconditional generation) than our work (conditional inference). All mathematical and algorithmic notations are the same as those in the main manuscript.
In the supplementary video, we provide overview of our paper.
B IMPLEMENTATION
Here, we provide implementation details of our uORF model.
B.1
OBJECT-CENTRIC LATENT INFERENCE
We show a pseudo code of inferring object-centric latents with the background-aware slot attention in Algorithm 1.
Convolutional feature extraction. The convolutional net extracts features from the input image for updating the latent slots. Our convolutional encoder is a simple U-net. We show our encoder architecture in Table 8 and Table 9. Since we want the model to generalize decompose unseen images, it is natural to represent foreground objects position and pose in the viewer coordinate system. As identified in previous studies (Tatarchenko et al., 2019), this facilitates the learning of 3D object position and helps generalization. In order for the object-centric representations to include such information in the viewer coordinate system, we can inform the encoder of position information by feeding pixel coordinates and viewer-space ray directions experiments we assume fixed camera focal length. In this case, the ray direction does not provide additional information the pixel coordinates, and thus we only feed pixel coordinates input channels in addition to the input RGB image. Each of the XY pixel coordinates is normalized to [-1, 1] in both directions, leading to 4 additional channels RGB.
additional input channels. In
B.2 COORDINATE SPACE AND LOCALITY CONSTRAINT FOR BETTER FORE-/BACK-GROUNT DISENTANGLEMENT
Coordinate space. We represent foreground objects in the viewer space. Regarding background environment, we represent the world coordinate space for two reasons. Firstly, since it difficult to estimate full geometry from a single view (e.g., the geometry behind the camera), our model assumes a similar background geometry across scenes and aggregates information about background geometry from multiple sparse views. Representing background in fixed world space facilitates this aggregation process and empirically leads to better performance. We show quantitative comparison in Table 10, Table 11 and visual comparison in Figure 7. Modeling the background in world space provides more details than modeling it in viewer space. Incorporating multi-view images as inference input might relax this assumption (Yu et al., 2020), but leave it future exploration.
13

---

Published as conference paper at ICLR 2022
Layer name
Input shape
Output shape
Stride
Note
Conv1
64x64x7
64x64x64
Skip to Conv6
Conv2
64x64x64
32x32x64
Skip to Conv5
Conv3
32x32x64
16x16x64
Conv4
16x 16x64
16x 16x66
Upsample
16x 16x64
32x32x64
Bilinear upsampling
Conv5
32x32x 128
32x32x64
Upsample
32x32x64
64x64x64
Bilinear upsampling
Conv6
64x64x122
64x64x64
Table Encoder architecture for the CLEVR-56 dataset and the Room-Chair datasct. convolutional keree sizes are 3x3. activation functions convolutional layers are ReLU.
Layer name
Input shape
Output shape
Stride
Note
Conv0
128x x28x7
22x118xx6
Conv1
128x128x66
64x64x6A
2
Skip to Conv6
Conv2
64x64x64
32x32x64
2
Skip Conv5
Conv3
32x32x64
16x16x64
Conv4
16x16xx4
16x16x64
Upsample
16x16x64
32x32x64
Bilinear upsampling
Conv5
32x32x 128
32x32x64
Upsample 32x32x64
64x64x64
Bilinear upsampling
Conv6
64x64x 128
64x64x64
Table 9: Encoder architecture for the Room-Diverse dataset. All convolutional kernel sizes are 3x3. All activation functions for convolutional layers are ReLU,
Secondly, this design also encourages the disentanglement between foreground objects and back- ground by preventing the background slot from decoding foreground objects, because the positional information provided in the encoder is represented in viewer space.
Foreground locality. To further encourage the disentanglement, we add locality constraint during early training prevent foreground slots to represent the background environment. Specifically, considering that "foreground" objects should largely visible in sight, we set a foreground box and enforce that every foreground-queryinn box is defined such that its projection in image space can engage roughly 90% pixels. The locality constraint is imposed for the first 100K iterations, and it empirically helps prevent the foreground slots from fitting the background. We show a visual comparison Figure 8, which from we can observe that the model without foreground locality design attaches some background segments to each object.
point outside the box has zero density. The foreground
B.3
NEURAL RADIANCE FIELD ARCHITECTURE.
We show our conditional object radiance field architecture in Figure
B.4 MODEL LEARNING
Loss functions. We set Apercepi = 0.006, Aui, 0.01, An = 10. For perceptual loss, we implement the feature extractor by using the output the 4-th convolutional block a VGG16 (Simonyan & Zisserman, 2014) pretrained on ImageNet. For the adversarial discriminator, we follow the architecture of StyleGAN2 (Karras et al., 2020) with slight modification such that the maximum channel number is 128. We also use the lazy R1 regularization (Karras et al., 2020). We Adam optimizer for discriminator with learning rate 0.001, 8 = and Br = 0.9. The adversarial loss incorporated after 100K iterations. Since shape uncertainty only appears in the Room-Diverse
14

---

Published as conference paper at ICLR 2022
Algorithm 1: Object-centric latent inference with background-aware slot attention.
Input: feat E DR.
Learnable: 1r a0,a" .uf, ( o prior parameters, h ,", ,.." ob, vf: linear mappings, GRU", GRU MLP", MLP!
slot" ~ N² RIID
// Sampling slots from priors.
slots/ N¹1 E RRXD
fo 1........>
slot prev" slot",
slots prev'
= slots
{ (slot") 4 (slots')
attn Sof tmax
1 (feat))
dim='slot?
// Binding slots to object features.
VD
attn" attn [0], attn attn[1:en
updates ==
values=u (inputa)) vaapes= // Aggregating update signals.
updates/= WeightedMeen (veights=attm', values=y/ (inputs))
slot GRU (state=slot.prre" inputs=updates")
// Updating slots.
slots' GRU' (state inputs=updates"
slot 6 + MLP (slot"), slots/ + MLP/ (slots')
// Residual update.
return slot", slots!
3D metric
2D metric
Models
Models
LPIPS
SSIM
PSNRT
NV-ARIT
ARM
Fg-ARI1
uORF siew-space Backg. dORF (ours)
0.799
27.86 86
VORF view-space Backg.
73.5
78.0
89.0
0.082
0.872
29.60
OORF (ours)
74.3
78.8
88.8
Table Ablation for background coordinate space on novel view synthesis on Room- Room-Chair dataset.
Table II: Ablation background coordinate space on segmentation on Koom-Chati dataset.
dataset, we only impose the adversarial loss on the Room-Chaii Both perceptual loss and adversarial loss are added after the first 100K iterations.
dataset but not on CLEVR-567 or
Coarse-to-fine progressive training. For coarse training, we bilinearly downsample supervision images to 64x64. The coarse training lasts for 600K iterations. For fine training, we randomly crop 64x64 patches from 128 x 128 images. The fine training lasts for 600K iterations. Our model trained on a single Nvidia RTX 3090 GPU for about 6 days. For all networks except discriminator, we use Adam optimizer with learning rate 0.0003, 31 = 0.9 and 3g = 0.999. Learning rate is exponentially decreased by half for every 200K iterations until after 600K iterations. We also adopt the learning rate warm-up from the slot attention paper (Locatello et al., 2020) for the first 1K iterations. We initialize decoder networks with Xavier's initialization. In each batch, input one image and neurally render 4 images for supervision. We render each pixel with 64 samples.
C EXPERIMENTS
In this section we provide further details on experiment settings.
C.1 DATA
For the construction of all three datasets, the training/testing sets share the same pool of textures, shapes, and colors. The scenes in both sets differ in the spatial arrangement of objects, as well as the appearance differences induced by soft shadows and inter-reflections due to global illumination effects.
CLEVR-567. In the CLEVR-567 dataset, each object's shape is randomly chosen from three geometric primitives (i.e., cylinder, cube and sphere). The color is randomly chosen from (red, blue, purple, gray, cyan, yellow, green, brown). There are two possible sizes for each object. When rendering images, son et al., 2017). We do not use the visibility check due to our 360 degree multi-view setting, so we increase elevation angle by π /5 115t to increase the chance of object visibility. Rendering setting the same for all datasets.
use the same camera intrinsic original CLEVR dataset (John-
For CLEVR-567 dataset set the latent dimension D 40 and the maximum number of objects K = 8.
15

---

Published as conference paper at ICLR 2022
Input view
Novel view
View-space
View-space
Ground truth
uORF (ours)
Ground truth background uORF (ours)
background
Figure 7: Visual comparison for representing background on view-space on novel view synthesis.
Input
view
Novel
view
Input view
Novel view
Ground truth
Ground truth
foreground uORF (ours)
Color image
segmentation
box
Figure 8: Visual comparison on ablation for foreground locality constraint. We show examples in CLEVR-567 testset. We can see that our foreground locality box helps prevent object slots from fitting background segments.
Room-Chair. For the object shape we use chair model from ShapeNet (Chang al., 2015). We use the same material and colors as CLEVR-567. For Room-Chat and Room-Divers. datasets, we set the latent dimension
and the maximum number of objects 5.
Room-Diverse. All object shapes are randomly chosen from 1,200 ShapeNet chairs. For each shape, we normalize it into unit cube according to vertex coordinates. We also use 8 colors (red, blue, purple, gray, cyan, yellos green, black) with diffuse material. Since shape uncer- tainty only appears in this dataset, we only impose the adversarial loss this dataset.
C.2 BASELINE ARCHITECTURES
Slot attention. We use the encoder-decoder architecture in the slot attention paper (Locatello et al., 2020) used for object discovery experiments on the CLEVR dataset. Basically it has convolutional layers for encoder and 6 convolution-tranne transpose layers for decoder. The number of channels for each layer is 64, All models are trained on 128 x 128 images.
NeRF-AE. We follow NeRF implementation without view direction as input and set the highest frequency to 5. The encoder is similar to ours in Figure 9, but the basic number of channels is increased from 64 to 256 (and thus the number channels of inputs to Conv5 and Conv6 is 512). The number of slot is 1.
D ADDITIONAL RESULTS
Generalization to unseen objects. the main paper we demonstrate systematic generalization to unseen combination of shape and color, here we further validate our model's generalization to unseen
Model ID: Br44774455002254422cc00000022229
16

---

Published as conference paper at ICLR 2022
(xx
Y
33
+
64
64
64
64
64
64
64
>
64 ...
RGB
z
64
:input
output
hidden states
+ vector concatenation
>linear w/ activation
linear w/ Sigmoid
linear w/ ReLU
Figure 9: Illustration for foreground decoder architecture. We follow the architecture in NeRF (Mildenhall et al., 2020) but with fewer parameters to decrease space demand. We set the highest positional embedding frequency to 5. so that the positional embedding input dimension is 5 x 2 x + 3 33. The background decoder is slightly diffcrent in that does not have the second last layer and third last layer. Density a activated by RcLU. Since estimating specularity from a single image intrinsically ambiguous, we assume Lambertian surfaces and do not the ray direction as input.
3D metric
2D metric
Models
Models
LPIPS
SSIM
PSNR
NV-ARIT
ARIT
Fg-A
uORF shape
0,1729
0.7054 0.7125
25.96
WORF shape testset
56.9
65.6
67.9
uORF on unseen shupe testsel
0.1771
26.16
wORF on unseen shape testset
$7.0
66.1
67.7
Table 12:
Novel view synthesis results on unscen/seon shape testset of Room-Diverse.
Unsupervised segmentation 3D results on Table 13:
unseen/seen shape testset of Koom-Diverse.
object shapes. To this end, we construct another test set for Room-Diverss All test objects in the new test set are drawn from pool of 500 shapenet chairs that are completely disjoint from the 1200 training chairs. All other settings are the same as the original test set. We show quantitative results in Table 12 for novel view synthesis and in Table 13 for segmentation. As we can see, our model yields the same level performances even on unseen shape test set, suggesting its generalization to unseen object shapes.
Generalization to real images. We also take a step further to test our pretrained model's general- ization real photos. To do this,
uORF trained on Koom-Diverse We take few real photos by a cellphone, providing an input image and a few reference images. We show the visual results in Figure 10. Although the real photo has a different imaging process and consists of unseen objects and background, uORF is able to discover all objects with roughly correct positions and orientations, yielding plausible segmentation results and object-moving results.
Analysis on the sensitivity to slot initialization. We test the robustness of our model to the slot initialization on the Room-Chan dataset. For each test scene, we now use 5 different random seeds for sampling initial centers. We compute the mean μ and std of ARI over the 5 seeds. We average them over the 500 test scenes. The averaged mean f of ARI is 78.8% and ō is 1.7%. The mean ARI suggests good segmentation results (very close 78.8% reported in Table 2 in main paper), and ō = 1.7% indicates that different seeds all lead to results close to such good ARI performance.
Additional qualitative results. We show additional qualitative results for our experiments in the main manuscript. We show additional examples for scene segmentation in Figure 11 and Figure 12, for novel view synthesis in Figure 13, Figure 14 and Figure 15, for scene editing in Figure 16 and Figure 17, for evaluating losses in Figure 18, for generalization to challenging spatial arrangement Figure 19 (note that the packed-CII CLEVR-II dataset we only use a single size for higher object visibility), and for generalization to unseen object appearance in Figure 20.
Failure case. our experiments, we observed type of failure which we call "attention rank- collapse". We show examples in Figure 21.
17

---

Published as conference paper at ICLR 2022
Input view
Novel
view
Reul input photo
Real photos
Ours, ro-rendering Segmentation
Moving object
Ours, moving object
Figure 10: Demonstration generalization real photos. We use uORF pretrained Room-Diverse and take photos by cellphone.
"Attention rank- cank-collapse" refers to that all the foreground object slots have (nearly) the same attention map and collapse to the same representation. Each collapsed slot decodes simply nothing (or all the foreground objects). This "attention rank-collapse" happens when the initialization is prompt to degenerate solution for the slot attention. It occasionally happens and empirically changing the initialization seed can address it. A related rank-collapse problem is discussed Dong al. (2021), which suggests that adding some architectural inductive bias can largely alleviate the problem. We hope future research can address this problem fundamentally.
E COMPARISON GIRAFFE
Our work has fundamentally different focus compared GIRAFFE (Niemeyer & Geiger, 2020). While GIRAFFE focuses on unconditional generation and enables multi-object scene synthesis and rendering, the goal of our uORF simultaneously infer 3D multi-object scene representations from a single image, in addition to using those representations for rendering and editing as in GIRAFFE.
E. COMPARISONS BETWEEN OUR UORF AND GIRAFFE
To demonstrate that the inference of such multi-object scenes highly non-trivial, we compare with GIRAFFE on both CLEVR-567 and Room-Chair (we cannot compare their datasets because they only have a single image for each scene). To train GIRAFFE on our datasets, we use the official repo and the same hyper-parameters that GIRAFFE authors used for their CLEVR-234, dataset, except for a few adaptive changes to our datasets: (1) We try different sizes for the object slot, because CLEVR-2345 only uses small objects while datasets both contain larger objects. Specifically, we try 2x, 1.5x, and xx original size, and use the one with the lowest FID for each dataset. (2) We adjust the camera elevation angle and focal lengths match our datasets. (3) We set the number of objects to 4 for the Room-Chai dataset because each scene has more than 4 chairs. We train the GIRAFFE models for around 500K iterations on 128-by- 228-by-128 128 images, such that FID does not drop anymore.
For GIRAFFE to inference, we sample object (including background) latents and positions in the same manner as training, and then we optimize for L2 reconstruction loss for both the latents and the positions. We use Adam and do a learning rate sweep to select the one that leads to the best reconstruction loss. We divide the learning rate by 10 when the loss platcaus. We do this learning rate decay twice. We sweep in (0.1, 0.01, 0.001, 0.0001} and find that 0.01 works best. Since each scene set the number the maximum number across all scenes. It has an unknown number of objects, CLEVR-567 and around 300 iterations on Room-Chair. Thus converges at around 150 iterations on we set the maximum iteration 300 and 500 for them, respectively.
We also compare with a GIRAFFE model that is pretrained on CLEVR-2345. The pretrained model provided by the authors. The pretrained model yields FID= 82 CLEVR-567 (FID= 61 CLEVR-2345), indicating that it could be valid baseline even though the two datasets are mildly different.
We show input-view reconstruction and novel view synthesis results in Table 14 and Table 15, and we show qualitative comparison in Figure 22 and Figure 23. We can see that GIRAFFE fails in
con//autonooouvviiioo.o /tt. /graaffe
18

---

Published as conference paper at ICLR 2022
Input view reconstruction
Novel view synthesis
Models
LPIPS
SSIM
PSNR1
LPIPS
SSIM
PSNR1
GIRAFFE (trained CLRVR-5677 GIRAFFE model on CLEVR-2345) uORF (ours)
0.330
0.815
23.75
0.549
0.672
16.65
0.382
0.780
21.76
0,643
0.348
11.70
0.085
0.901
29.33
0.086
0.897
29.28
Table 14: Inference comparison with GIRAFFE on CLEVR-567.
Input view reconstruction
Novel view synthesis
Models
LPIPS|
SSIM
PSNR
LPIPSI
SSIM1
PSNRT
GIRAFFE (trained on Room-Chair)
0.414
0.597
20.90
0.588 0.082
0,538
18.53
uORF (ours)
0.085
0.876
29.65
0.872
29.60
Table 15: Inference comparison with GIRAFFE on Room-Chair.
reconstructing the multi-object scenes from a single image, as well as novel view synthesis. Let alone segmentation 3D.
E.2 GIRAFFE INFERENCE ON CLEVR-233 -2445
While we have compared with GIRAFFE on our datasets, we further evaluate the author-provide pretrained model on the simpler dataset CLEVR-236 from the GIRAFFE paper itself. We found that while GIRAFFE does well on unconditional scene synthesis, it cannot perform novel view synthesis on their own dataset, either. This shows that GIRAFFE focuses problems very different from ours.
We first show that inferenceerecoonstuuiion is challenging for GIRAFFE, even on the simpler dataset. We do inference the author- provided CLEVR-234 dataset using the author-provide pre- trained model. We show randomly sampled examples through the iterative inference process in Figure 24.
Then we show that GIRAFFE fails in wide-baseline novel view synthesis. We use the author- prooidddd pretrained model sample from its latent space and unconditionally generate image. Then keep all the variables the same, but circularly move cameras to render novel views. We show 10 random examples of this circular novel view synthesis in Figure 25. We see that when the viewpoint changes become significant, GIRAFFE fails novel view synthesis, because its neural renderer is based on 2D feature maps and it's not inherently 3D.
E.3 DISCUSSION AND SUMMARY
In general, inverting GAN latent space even for the holistic image non-trivial and needs architectural- specific designs (we refer the reader to the discussion and references in a recent survey on GAN inversion (Xia et al., 2021)). As for inverting compositional multi-object scenes, it becomes even harder due to ambiguous correspondences ("which slot corresponds to which object?"), number of objects ("how many slots should put?"), object position constraints ("there are two objects overlapping in the image, but they should not be overlapping in 3D"), optimization issues (e.g.. optimizing rotation is notoriously difficult (Zhou et al., 2019)), etc.
In summary, it is highly non-trivial for GIRAFFE to do inference for multi-object scenes due to complexities such as ambiguous correspondences, the number objects, and optimization issues. We the difference between the two methods in the following separate
will include more discussions thread. In short, our uORF tries to solve a fundamentally different problem from GIRAFFE, i.c., we aim at inferring the joint distribution of objects from single image while GIRAFFE targets extrinsic- controllable image generation. Therefore, our method enables novel tasks such as unsupervised segmentation and editing in 3D, where prior methods including GIRAFFE are not able to do.
19

---

Published as conference paper at ICLR 2022
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel
N/A
Input
Novel
N/A
view
Input view
Novel view
N/A
Ground truth Ground truth segmentation
Slol Attention wlo background w/o prog. train.
uORF (ours)
Color image
Figure 11: Additional qualitative results for segmentation in 3D Room-Chair dataset.
20

---

Published as conference paper at ICLR 2022
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Ground truth Ground truth segmentation
Slot Attention win background wio prog. train.
nORF (ours)
Color image
Figure 12: Additional qualitative results for segmentation in 3D on Bhoo--Diverse dataset.
21

---

Published as conference paper at ICLR 2022
Input
view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel
view
Ground muth
NeRF-AE
w/o background
w/o prog. train.
uORF (ours)
Figure 13: Additional qualitative results for novel view synthesis CLEVR-567 dataset.
22

---

Published as conference paper at ICLR 2022
Input view
Novel view
Imput view
Novel view
Inpur view
Novel view
Input view
Novel view
Input
view
Input view
Novel
view
Ground
w/o background
w/o prog. train.
(ours)
Figure 14: Additional qualitative results for novel view synthesis on Room- -hhai dataset.
23

---

Published as conference paper at ICLR 2022
Input
Novel
Input view
Novel view
Input view
Novel view
Input
Input view
Novel view
Input view
Novel
view
Ground truth
NeRF-AF
wia background
w/o prog. train.
uORF (ours)
Figure 15: Additional qualitative results for novel view synthesis Room-Diverss dataset.
24

---

Published as conference paper at ICLR 2022
1. Move objects
Inpur view
Novel
N
Input image
view
background wio prog.
(ours)
II. Chunge background
Inpat view
Novel
N/A
Background image (input image the same above)
view
Ground truth
Slot Attention
NeRF-AE
wio background wio prog. train..
uORF (ours)
Move objects
Input
view
Novel N/A
Input image
view
Ground truth
Slot Attention
NcRF-AF
background prog.
nORF (ours)
II. Change background
Input
Novel
N/A
Background image (imput image
view
same above)
Ground truth
Slot Attention
NeRF-AE
wiO background wi prog. train.
EORF (ours)
Move objects
Input
view
Novel N/A
Input image
view
Ground truth
Slot Artention
NeRF-AF
wrfo background wio prog. main.
uORF (ours)
II. Change background
Inpat
view
Novel
N/A
Background image (inpat image the same as above)
view
NeRF-AE
background wio prog.
(ours)
Figure 16: Additional qualitative results for scene editing.
25

---

Published as conference paper at ICLR 2022
Move objects
Inpur view
Novel N/A
Input image
Ground truth
Slot Artention
NeRF-AE
wro background wro prog. train.
wORF (ours)
IL. Chunge buckground
Input
Novel
N/A
Background image (input image the same ahove)
view
Ground truth
Slot Attention
NeRF-AE
wio buckground w/o proy. truin.
uORF (ours)
Move objects
Input
N/A
Input image
view
Ground truth
Slot Attention
NcRF-AE
background wio prog.
uORF (ours)
Change background
Input
view
Novel
A/A
Background image (input image the above)
view
Ground truth
Slot Attention
NcRF-AE
wio background w/ prog. train.
uORF (ours)
Move objects
Input
view
N/A
Input image
Ground Attention
background wio prog.
(ours)
Change background
Inpur
view
Novel
N/A
Background image (input image the above)
view
Ground truth
Slot Attention
NeRF-AF
w/o backeground w/o train.
#ORF (ours)
Figure 17: Additional qualitative results for editing.
26

---

Published as conference paper at ICLR 2022
Ingut view
Novel
Input view
Novel view
Input view
Novel view
Input
Novel view
Input view
Novel view
Input
Novel view
Ground truth
Recon.
Recon. adv.
Reson. percept.
vORF (ours)
Figure 18: Qualitative results for loss evaluations. Using both perceptual loss and adversarial loss improves image quality.
27

---

Published as conference paper at ICLR 2022
Input
view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Input view
Novel view
N/A
Ground truth Color image
Ground truth segmentation
Slot Attention
uORF (ours)
Figure 19: Qualitative results for generalization spatial arrangement.
28

---

Published as conference paper at ICLR 2022
Input
view
Novel
N/A
view
Input view
Novel view
N/A
Input view
Novel view
N/A
Ground truth Color image
Ground truth segmentation
Slot Attention
uORF (ours)
uORF (oracle)
Figure 20: Qualitative results for generalization to unseen combination of color and shape.
Input
view
Novel
view
Re-rendered (Backkgroud slot only)
Re-rendered object slot only)
Ground truth
Re-rendered
Figure 21: Failure case of our model, which we call "attention rank-collapse". All foreground slots share the same attention map. Every foreground slot decodes to the same radiance field (empty radiance here) rather than specializing to an object. Here we only show one object slot, as all others look the same.
29

---

Published as conference paper at ICLR 2022
Input view
Novel view
Input view
Novel view
GIRAFFE (pretrained by authors)
GIRAFFE
Ground truth
(trained CLEVR-567)
uORF (ours)
Figure 22: Visual comparison with GIRAFFE for inference CLEVR-567 dataset. GIRAFFE fails inference.
Input
view
Novel
view
Input view
Novel
view
Ground
GIRAFFE
(ours)
Figure Visual comparison with GIRAFFE for inference on Room-Chair dataset. GIRAFFE fails inference.
30

---

Published as conference paper at ICLR 2022
Initialized genemntion
Targer image infer
iterations
lterations
iteaaiioss
Figure 24: Inference trajectory of GIRAFFE using author author-provided models on the author protodr dataset CLEVR-2345.
31

---

Published as conference paper at ICLR 2022
camera
geserated
Rotate camera by a80
Rotate camera by 157.0"
Rotate camera by "25"
Rotate camera by "112.5
Rotate camera by "yu"
Rotate camera by -67.5"
Rotate camera by "4."
Rodate camera by -22.5"
gemersted
45"
69
by
by 122."
135"
by 137
Rotate cameers by 157.5"
Rotate camerr by -35
camera -11.5
camera -00
camera by -6 5
camera by -45
camers by -22.5
by 180
Initial generated
Rotatt comera by 22.5"
Rotate tamera by
Rotate camera by d*
Rotate camera by
Rotate camera by 122.5
Rotate camera by 135"
Rotate comera by 137
by
by
Figure 25: Novel view synthesis on randomly generated examples using author-provided pretrained GIRAFFE model on CLEVR-2345 GIRAFFE fails inference of these multi-object scenes. GIRAFFE cannot synthesize novel views with large rotations.
32


Reviews:
[{"review_id": "qZY0RKnsnLp", "reviewer": "Reviewer_BYk3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n\nThe overall direction is promising, and factorize the scene representation is indeed an important issue to study. \nThe technique proposed is sound overall with soft-kmeans like strategy to generate corresponding features in an unsupervised manner, through the GRU probably break the theoretical guarantee of convergence. \n\nWeak: \n\nThe overall results looks more in a concept proof, the objects in all test datasets are relatively simple having uniformed color. Feel it can hardly work in real scene senario, as shown in GIRAFF paper.  Under these senarios, the segmented results and rearranged results lose many object details yielding blurry or incorrect renderring.  I feel there should be more improvement over these issues. \n\nThe overall concept is fairly close to GIRAFF and major difference could be the training scheme inference from a single image or multiple. I would like to see there could be additional techniqual improvement especially some high resolution representations. Or improvement of architectures in order to support better quality. \n\nsome questiions: \nDoes the algorithm always obtains a reasonable representation w.r.t different initial sampled centers ? \n\n\n\n", "summary_of_the_paper": "The paper proposes a novel neural representation of an given image, which decomposes K object instances from background. So various tasks such as rerendering, rearrangement etc. \n\nThe learning process is first sampling K centers, and represent each object as a learnable hidden variable z,   an gaussian based soft. k-means styple clustering is then performed afterwards. The differences here are 1)  for sampling the centers, there are  learnt forground and background priors, which can benefit the initial state in this learning process.  2)  updating the centers z with a learnable GRU rathor than simple mean pooling, which I believe it have more flexible representations for the cluster. Finally,  the object clusters are discoverred. In this process. \n\nThe author evaluted with 3 self created datasets and show several reasonable results by performing mentioned tasks such as 3D segmentation, rearrangement etc. \n", "summary_of_the_review": "This paper points out a good direction to dive into unsupervised learning of compositional scene representations. However, technique strongness and novelty may need to be further improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hRRR-ywWmNC", "reviewer": "Reviewer_tkSe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors presented a novel idea. The system is engineered well and the authors have shown success on three synthetic datasets and various applications.\nMy concerns are as follows.\n1. All experiments are conducted on synthetic datasets. Both training and testing use the same set of object shapes - only the arrangements of the objects are different. It is not clear how this approach can generalize to real world scenes. As in the real world, lots of objects have not been seen in the training set. An ablation study that adds unseen shapes into the testing scenes can be very informative. Additionally, a demonstration of the approach on real world scenes would be a strong result to show in the paper (either it is negative or positive).\n2. How is the number of the foreground objects decided? Does it have a strong impact in the results?\n\nEdit: The authors addressed my concerns in their revision. In particular, the authors showed additional results on a real world image. As expected, the rendering is not as good as on synthetic data. However, I do not think this overshadows the contribution of this paper - instead, it shows the value as well as limitation of the proposed method, and can inspire future work.", "summary_of_the_paper": "This paper utilized the powerful NeRF. The authors present a new approach to learn the scene arrangement in an unsupervised way. The training is performed on unlabeled datasets of similar objects in different arrangements. The inference requires only one RGB image as input, and can correctly deduce the arrangement and the 3D geometry of the objects. The authors showed two supporting technical contributions to the system: (1) splitting background and foreground objects leads to better results, (2) a coarse-to-fine training to alleviate the space and time needs. The authors showed success on three synthetic datasets and various applications.", "summary_of_the_review": "The authors presented a novel idea. The system is engineered well and the authors have shown success on three synthetic datasets and various applications. However, all evaluation and experiments are done on synthetic datasets with the same set of objects and background. Thus it is unclear how this approach can generalize to solve real world problems.\n\nEdit: The authors addressed my concerns in their revision. I think it is a good paper and should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cD0wWdGvyMO", "reviewer": "Reviewer_pN74", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##################################################################\n# Pros\n1. The authors address a significant new problem: modeling 3D scenes as a disjoint set of objects that can be combined and rendered for novel view synthesis. Their model can also be learned from only 2D data.\n\n2. The proposed method uses state-of-the-art techniques to achieve its goal. Namely, it uses slot-attention mechanisms (NeurIPS 2020) and Neural Radiance Fields (ECCV 2020) and combines them to address a new problem.\n\n3. Treating background and foreground latent vectors as being drawn from two separately learnable distributions addresses one of the significant drawbacks of slot attention.\n\n4. I appreciated the authors mentioning that concurrent work by Stelzner et al. (2021) addresses the same issue and differentiates this paper appropriately. \n\n5. The proposed method is overall technically sound, and code is provided, which will help with reproducibility. Also, the authors do an excellent job at mentioning all the hyper-parameters and model architecture details as far as I can see.\n\n6. The paper provides comprehensive experimental evaluations, both quantitative and qualitative.\n\n####################################################################\n# Negatives / Questions\n1. It would have been great to compare this work to the concurrent work of Stelzner et al. (2021). I believe that this would help the community to put the two concurrent submissions into context. That said, I do recognize that the work of Stelzner et al. (2021) has not been published and that code/data for their approach is not publicly available, making comparisons extremely difficult and should therefore not be a requirement for publication of this work. I hope that future work in this direction will pick up this issue.\n\n2. Section 3.3, Coarse to fine training. I agree that rendering images with the volumetric rendering framework proposed in NeRF requires many evaluations per ray, so reducing the number of rays sampled during training makes sense. One detail that is missing in this section is how many samples per ray are used? Moreover, do the authors still use the two networks for ray evaluation as NeRF (coarse and fine ones)? If not, why not? Second: In your approach, you sample random patches during the fine training stage and downsample the images during the coarse training stage. In this paper: \"GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis,\" the authors propose a different sampling strategy, which does not require downsampling. Would this strategy perform better? Worse?\n\n3. Section 4.1, Segmentation Experiment Results. It would help to discuss the shortcomings of slot-attention, namely, why the method fails on the segmentation task. My guess why uORF works better would be due to the two disjoint latent spaces for foreground and background? I think this should be highlighted here.\n\n4. Section 4.3, Scene Design and Editing Setup. I did not understand how the setup for modifying the foreground object pose/appearance works. I think you can switch the latent embeddings for background, as you have a one-to-one mapping, but this is not the case for the foreground objects. Could you explain this process in more detail?\n\n5. Appendix B2, Coordinate Space. Here you mention that you use a foreground box to encourage disentanglement of foreground and background slots. How does this foreground box work, what is its influence on the final result? Mentioning this (maybe) crucial detail only in the appendix is not sufficient in my opinion and should be better explained in the main text.\n", "summary_of_the_paper": "The paper introduces an interesting new research direction of factorized, 3D consistent neural representations. In particular, it proposes to combine slot-attention mechanisms with conditional neural radiance fields to segment and render novel views of a scene from a single input view. The authors also address one apparent shortcoming in the slot-attention paper: the background and foreground object latent codes are sampled from the same distribution, leading to breakdowns on scenes with complicated backgrounds. The paper proposes to learn two disjoint distributions, one for the background and one for the foreground, to alleviate this issue.", "summary_of_the_review": "I vote to accept this paper for publication at ICLR 2022. I like the idea of modeling scenes as a combination of disjoint objects, which can be added, removed, modified, and recombined to form new scenes. I also think the paper is well written, well-motivated, and provides extensive experiments. In my opinion, the paper adds to the literature on neural scene representation/decomposition and is interesting to the community. I have some minor suggestions and questions (see above), which I hope the authors clarify during the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a method for decomposing scenes into object-specific neural radiance fields.  After the discussion phase, two reviewers support acceptance.  Empirical results on multiple synthetic datasets and benchmarks appear convincing; the rebuttal also added an initial demonstration of generalization to real images.
Decision: Accept (Poster)