Unsupervised Discovery and Composition of Object Light Fields
Authors: Cameron Omid Smith
Hong-Xing Yu
Sergey Zakharov
Fredo Durand
Joshua B. Tenenbaum
Jiajun Wu
Vincent Sitzmann
Venue: Accepted by TMLR ()

Abstract: Neural scene representations, both continuous and discrete, have recently emerged as a powerful new paradigm for 3D scene understanding. Recent efforts have tackled unsupervised discovery of object-centric neural scene representations. However, the high cost of ray-marching, exacerbated by the fact that each object representation has to be ray-marched separately, leads to insufficiently sampled radiance fields and thus, noisy renderings, poor framerates, and high memory and time complexity during training and rendering. Here, we propose to represent objects in an object-centric, compositional scene representation as light fields. We propose a novel light field compositor module that enables reconstructing the global light field from a set of object-centric light fields. Dubbed Compositional Object Light Fields (COLF), our method enables unsupervised learning of object-centric neural scene representations, state-of-the-art reconstruction and novel view synthesis performance on standard datasets, and rendering and training speeds at orders of magnitude faster than existing 3D approaches.

Published in Transactions on Machine Learning Research (05/2023)
Unsupervised Discovery and Composition of Object Light Fields
Cameron Smith', Hong-Xing Xing Yu², Sergey Zakharov Fredo Durand', Joshua B. Tenenbaum' 1.4.5 Jiajun Wu², Vincent Sitzmann
'IIT CSAIL Stanford University
Toyota Research Institute
*MIT BCS
CBMM
canerorosmiih..ituubbuu
Reviewed on OpenReview:
Abstract
Neural scene representations, both continuous and discrete, have recently emerged as powerful new paradigm for 3D scene understanding. Recent efforts have tackled unsupervised discovery of object-centric neural scene representations. However, the high cost of ray- marching, exacerbated by the fact that each object representation has to be ray-marehed separately, leads to insufficiently sampled radiance fields and thus, noisy renderings, poor framerates, and high memory and time complexity during training and rendering. Here, we propose represent objects in an object-centric, compositional scene representation as light fields. We propose a novel light field compositor module that enables reconstructing the global light field from set of object-centric light fields. Dubbed Compositional Object Light Fields (COLF), our method enables unsupervised learning of object-centric neural scene representations, state-of-the-art reconstruction and novel view synthesis performance on standard datasets, and rendering and training speeds al orders of magnilude faster than existing 3D approaches.
1 Introduction
A critical aspect of scene understanding is parsing the scene into its composite parts. On the highest level, these are the static scene elements as well as rigid objects. It is this scene decomposition that enables us to quickly understand and subsequently interact with our environment, and is relovant to downstream tasks ranging from robotics to autonomous navigation to generative modeling.
A recent exciting avenue of research leverages generative modeling to learn scene decomposition in an unsupervised manner. One line of work accomplishes this by modeling objects in 2D images (Eslami et al.
2016 Crawford & Pineau 2019
Kosiorek al.) 2018 Lin et al.] 2020 Jiang et al. 2019 Burgess et al. Greff et al. 2019: 2016 2017 Engeleke et al.; 2019 Locatello et al. 2020). However, these models lack critical aspect of scene understanding, which is the reconstruction of the underlying 3D structure. A recent line work has instead proposed to model objects directly in 3D via object-centric neural scene representations (Yu al. 2021 Stelzner et al. 2021). Leveraging differentiable rendering, these models can be trained just from 2D image observations, offering exciting new perspectives on the unsupervised lcarning of visual representations useful to downstream lasks such as robotics, tracking, and scene understanding.
2019:
While promising, fundamental limitation of these models is that they suffer from the high computational complexity of volume rendering. Rendering images from a single neural radiance field is computationally costly as it requires dense sampling of 3D points along each ray (Mildenhall et al. 2020). Rendering of object-centric representations is more expensive yet, as each object radiance field needs to be ray- marched soparately and rendering cost thus scales with the number of objects in the scene. This high computational complexity prevents existing methods from achieving real-time rendering, and further limits the number of objects in each scene to about 10 objects. Tt further limits the number of samples per ray, leading to

---

Published in Transactions on Machine Learning Research (05/2023)
Slot Attention
Object LFNs
Light Field Compositor
Pái
Object Codes
Color, Ordering
Unorderd
Ray-Orderd
Query View
OBC)
Context View
Figure 1: Overview. We represent scenes Compositional Object Light Fields (COLF). Given image, we first use slot attention to infer a set object codes describing delected objects in the scene. Each object code queried for a color, well as ordering value (related to the camera depth) observed at the first intersection of the object and novel ray r. Our Light Field Compositor module then determines ray-visibility weights and yields color observed by
low-quality renderings due insufficiently sampled object radiance fields. Though recent progress has been made in speeding up volume rendering in the case of reconstructing single scene, applying similar techniques to the domain of generalization and prior-based reconstruction is an open problem.
In this work, we overcome this limitation. We propose modeling scene not composition of 3D elements, but as a composition of neural object light fields. Rendering neural object light field only requires sampling the neural representation exactly once per ray. However, in contrast to volume rendering, compositing of object-centric light fields is not analytical, as their per-ray depth order is unknown. Thus, we propose a novel light field compositor module that learns to estimate soft visibility for each object ray query, enabling compose a set of object-centric light. fields into the light. field of the complete scene. We demonstrate that our method not only enables critical speed-up and memory reduction, but also outperforms previous statc-of-tho-art methods in terms of reconstruction quality, and enables editing to compose and render unbounded scenes with tens of objects.
In summary, we make the following contributions:
We propose novel method that leverages neural light fields for the unsupervised discovery of objects.
We address the challenge of non-analytical compositing of partial light fields by proposing paramctric light field compositing module.
Our method significantly reduces the memory requirement and time complexity, enabling real-time rendering at state-of-the-art reconstruction quality.
2
Related Work
Neural Scene Representation and Rendering: Our method related recent work on inferring latent parameters of 3D scenes from images. Eslami et al. (2018) proposed encode several image observations of scene into latent code that subsequently be decoded into novel views with a convolutional neural network. 3D voxelgrids, combined with differentiable ray-marching, first allowed self-supervised discovery of shape and appearance from images (Sitzmann et al. 2209A]) Lombardi et al.) 2019). Inspired by neural implicit shape representations (Park et al. 2019 Mescheder et al. 2019 Chen & Zhang 2019), neural-field based representations, combined with neural rendering, lifted limitations of resolution (Sitzmann et al. 2019b: Niemeyer et al., 2020 Mildenhall al. 2020 Yariv al. 2020 Xie
(021). By conditioning on latent variables, this enables 3D reconstruction from just a single observation [Sitzmann et al. 2019b; Niemeyer et al. 2020). By conditioning locally on image features (Saito et al. 2019) and constraining ourselves to
2

---

Published in Transactions on Machine Learning Research (05/2023)
learning about local patches instead of object-centric or scene-centric representations, may generalize to scenes with unseen numbers of objects (Yu et al. 2020; Trevithick & Yang 2020). Most recently, light fields have been proposed as an alternative to 3D-ssructured representations to address the extraordinary cost of rendering (Sitzmann et al., 2021:: Instead of mapping a 3D coordinate to whatever cxists at that coordinate and thus requiring ray-marching for rendering, they directly map an oriented ray lo whatever observed by that oriented ray, therefore rendering with a single evaluation of the neural network per ray. However, these approaches do not infer object-centric representations, lacking semantic 3D scene understanding.
2D Compositional Scene Representations: Deep-learning based inference of object-centric representations was first addressed by factorizing 2D images into 2D) components, either represented as localized object-centric
patches (Eslami et al.) 2016; Crawford & Pineau 2019
Kosiorek et al. 2018
Lin et al. 2020 Jiang et al.)
Greff et al. 2019 2016 2017 Engeleke et al.] 2019). Locatello et al.] (2020) proposed Slot Attention as an inference model for such object-centric representations, which we also rely on in the present work. Though slot attention has in the past been limited to toy scenes, recent, concurrent work has demonstrated extensions to more complex scenarios. Improvements include leveraging weak supervision in the form of optical flow and object bounding- boxes (Kipf et al. 2021), : geometry supervision in the form of depth maps (Elsayed et al. 2022), or attention-based instead of mextture-bssed decoding (Singh et al. 2022). Our work is orthogonal: we do not propose any improvements to the inference algorithm, but rather, propose an alternative object rerpresentation in the form of 3D object light fields, enabling 3D object-centric representation learning. Improvements in inference as discussed in this concurrent. work might thus also benefit. the proposed method.
2019) or scene mixture components [Burgess et al., 2019
3D Compositional Scene Representations: Recent work has addressed unsupervised 3D scene decompo- sition. Elich et al. (2020) infer object shapes from a single scene image, but require ground-truth shapes for pre-training. Chen et al. (2020) extend the Generative Query Network (Eslami et al. 2018) to decompose 3D scones, but require multi-view observations during inference. Bear et al. (2020) model a scene as a scene graph and infer parametric object-centric shapes. Block-GAN and GIRAFFE (Nguyen-Phuoe et al., 2020 Niemeyer & Geiger| 2020) build unconditional generative models for compositions of 3D-structured representations, but only tackle generation, not reconstruction. Closest to our method are [Stelzner et al.) (2021) and Yu et al. (2021p, p, who use slot-based encoders for unsupervised discovery of objects. However, both approaches use volumetric neural scene represenlations (Mildenhall et al. 2020 due to the severe cost. of rendering at both training and test time, these approaches cannot render at real-time frame-rates and require either single-digit batch-sizes (Yu et al.) 2021 or ground-truth depth to accelerate training (Stelzner et al. 2021), and even then suffer from rendering artifacts due to insufficient volumetric sampling. While recent work has achieved impressive progress in accelerating the reconstruction and rendering of radiance fields for single scenes (Müller et al. 2022 Fridovich-Keil and Yu et al.) 2022: Garbin et al., 2021 Neff et al, 2021), applying these principles to the regime of prior-based reconstruction from few observations is an open problem. That is, because these methods leverage sparsity- sparsity-based techniques, such as skipping empty space or using sparse data structures, directly inferring the scene structure in a feedforward inference setting is difficult or at least has not yet been demoustrated. Our method not only significantly outperforms prior object-centric approaches in terms of reconstruction quality, but also addresses the computation and memory complexity of volumetric rendering.
Layered Representations for View Synthesis: Prior works leverage the multiplane image scene representation, a set of fronta-parallel RGBA planes predicted at various depth values, for the task of novel view synthesis (Zhou et al. 2018 Srinivasan et al. 2019). While this line of work is similar in the spirit of combining scene decomposition and novel view synthesis, our decomposition semantic (into objects), whereas their decomposition is geometric (into depth planes), and our view synthesis supports full 360-degree scenes, whereas theirs only supports front-facing scenes.
Light Field Compositing: Some prior work has investigated the composition of light fields parameterized as multi-plane images (Mildenhall et al., 2019: Dall et al., 2019). Chen et al. (2006) generalize the image alphna-copoosiing operator to light fields. However, both of these techniques are incompatible with object-centric light fields, as they assume front-facing (i.e. not 360-degree) scenes, and, critically, that
3

---

Published in Transactions on Machine Learning Research (05/2023)
Color & Ordering
Ray-Visibility Assignment
Oa),
V
(OOnc minn OBCC)
On
Onc
Frontmost Intersection
Σ
O min (onne""" ...
min
Figure 2: Light Field Compositor. Our light field compositor reasons about the relative order of surface-ray intersections for each of the object light fields, producing a set of feftmax-weighhee visibility scores. We then composite the contributions from each object light field into single color while respecting occlusions. See Sec. 3.2 for details.
the depth order along ray is known. Alas, this is not the case when seene is described by set of object-centric light field networks that can be rendered from arbitrary 360-degree perspectives, which we address by introducing the light field compositor module.
3 Method
Our goal is to build and train model that, given a single image of 3D scene, can parse it into set of K object-centric, 3D-aware representations and enables us re-render from novel views. In Sec. 3.1 we review the light field networks (Sitzmann et al.) 2021) neural scene representation. In Sec. 3.2 we introduce our new light field compositor that enables rendering of 3D scene from novel view points via occlusion-aware compositing of K light. field networks. In Sec. 3.3 we describe our new end-to-end auto-encoder model, the encoder, the losses, and the training strategy that enables us to learn to decompose scenes into object light fields. Fig. gives an overview over the proposed model.
3.1 Light Field Networks (LFNs)
Recently, Sitzmann et al. (2021) proposed to represent a 3D scene by directly parameterizing its 360-degree
light field (Levoy Hanrahan 1996 Gortler et al. 1996, Adelson et al. 1991 via a neural network. A scene is then parameterized as an MLP & that maps every 6-dimensional oriented ray directly the color observed by that ray: ((r) = c. Instead of sampling + hundreds thousands of times as in 3D-structured scene representations, rendering in light field networks reduces to single sample of & per pixel, achieving dramatic speed-up and reduction in memory complexity of rendering. Since 4 as an MLP, is differentiable, it can be trained by minimizing the L2 loss between the rendered colors ((r) and the colors of the training image I(r), i.c., L = | (r) Il.) We may also learn a prior over light ficlds that enables reconstruction from only a single observation by generalizing over a set of 3D scenes, where each 3D scene represented by a latent code z, and conditioning the light field network on that latent code, which we denote as (r; (Sizzaan
et al. 2021
3.2
Compositing LFNs
We wish decompose 3D scenes into their object parts by representing each object via separate Light Field Network (LFN). A scene is then represented set of K LFNs & D2, 4g. To formulate differentiable rendering algorithm, we require function that maps from a given ray r to the color e observed by that ray. We consider scenes without transparency, where the color of a given ray is determined only by the first surface it intersects, and therefore, only by one of the K LFNs. We must thus formulate an algorithm that composites the set of LFNs according to their relative depth ordering, such that we yield the color e, of the object with index j exactly if that object was hit by the ray r first, occluding the color obtained from other LFNs.
4

---

Published in Transactions on Machine Learning Research (05/2023)
In the case of neural radiance fields (Yu et al. 2021 Stelzner et al. 2021), composition is an analytical, non-parametric computation. This is because volume rendering samples poinls along rey, and therefore is aware of the order of the densities along the ray. This allows an analytical alpha-compositing rendering function even in the case of compositing soveral object-centric reprosentations to a single scone representation (Yu et al. 2021 p. At first glance, an equivalent approach in the compositing of light fields might. have each object-centric light field map a ray to a color C; and an alpha value 04, and then alpha-composite the colors to obtain the final color of the ray, as previously proposed in Chen et al. (2006). However, this assumes that the order of the object light fields along the ray r is known. This is not the case: sampling an LFN does not yield depth, and thus, the relative order of the surfaces observed in each of the LFNs is unknown. While it is possible to compute depth via the derivatives of the light field (Sitzmann et al. 2021), this depth is sparse, and thus cannot be used for compositing at every pixel.
We thus propose a Light Field Compositor model please see Fig. 2 for an overview. First, we extend LFNs to encode additional information in each of the object light fields o that allows us to compute their relative ordering. Specifically, we now map each ray to a tuple of color o, and ordering value oz
D R
R X R 2.(r) = (c;, 0i)
We note that the ordering value is not equivalent to camera depth in the conventional sense. This is because the output of the LFN is invariant to the query camera's position along the query ray (Sitzmann et al., 2021]. Therefore, the standard notion of regressing a depth value which increases as the query camera moves further away from the scene intersection not possible, as the output of the LFN by design does not change as the query camera moves along that ray. Rather, the ordering value intuitively be understood as the signed distance of each light field's ray-intersection relative to the projection of a global reference point, such as the world origin or the context camera position, onto the query ray r.
Given a set of K light field networks and a ray r, we query each LFN to obtain color and ordering values (cs, o:) = (r). We then use the minimum of the ordering values Omin = min(o),.... 02) to map the i'th light field's ordering value a visibility value v, V(,,, Omin - o,), where V is a small MLP. The difference Omin - oj intuitively corresponds to how far behind the light field i's intersection from the first estimated intersection. The visibility scores are then softmax- normalized and used to determine the color contribution from each light field, allowing us to obtain the final ray color as weighted sum of the per-slot radiances:
exp(1:)
I'(r) = K - ci SK exp(v;)
(2)
We considered explicitly making the ordering values relative to the projection of the context camera position onto the query ray, achieved by subtracting from each ordering value the distance from the camera's projection point to the query ray origin. However, because such shifting would preserve the relative ordering-val value differences between objects, and given that the visibility scores are softmax-normalized the projection-based shifting is unnecessary.
Our compositor related to prior work on differentiable mesh renderers, such as (Liu et al. 2019 Kato et al. 2018), which use a handerafted distance kernel to assign higher visibility to the frontmost ray-intersoctiinn face than subsequent faces along the ray. These handcrafted kernels, however, are tuned for the depth range and amount of gradient support needed by the application (Liu et al., 2019). We instead employ a learned weighting kernel, implemented by the small network V. This learned weighting kernel allows the network to adapt to the depth range of each dataset, whereas a handcrafted kernel requires fine-tuning for these two considerations.
We visualize the predicted raw and minimum-suubrrateed ordering values on our city-block datasct in Fig. Note how in the minimum-subtractt ordering value plot, the subject of each slot has a value of zero and all other subjects have a value below zero, yielding easier task for the occlusion kernel.
5

---

Published in Transactions on Machine Learning Research (05/2023)
FG
FG
BG
FG
Input
RGB
Imput
RGB
Novel
Visible
Novel
Visible
Input
RGB
Imput
RGB
Novel
Visible
Novel
Visible
Figure 3: Qualitative Scene Decomposition Results. We show light field decompositions four scenes. For each scene, the leftmost column shows the model input (top) and one novel (bottom) view. the right each scene, the first shows the RGB prediction from set of slots, and the bottom row shows the corresponding visibility mask, which filter out occluded rays prior to their composition into the final image. Please see the supplemental material for video results.
3.3 Learning to Represent Scenes as Sets of LFNs
We now describe a full decoder architecture that will enable us to learn, in an unsupervised manner, model capable of decomposing a scene into a set of object-centric Light Field Networks (LFNs) given a single image. An overview can be seen in Fig. I The encoder module will map a context image Ietat to a set of K latent codes 21, 22, .., Zg. Fach of these latent codes will be used to condition an object-centric light. field network (x(r) (ir; defined in the coordinate frame of the context image Terst. Following (Yul et al. 2021), further introduce background light field O BC(r) Φ(r; B whose latent is inferred by separate encoder, and which is defined in canonical world coordinate frame. Given query image Iquery of the same 3D scene, we parameterize the per-pixel camera rays via the explicit and implicit camera parameters, and sample each (r) and BC((r to yield set of colors and ordering values (C1, 01), (C2, o2), (......... (CK: ok) and background values (CBG, OBG). Finally, we leverage our compositor module to render a color for each ray, compute a reconstruction loss, and backpropagate that loss to train our model end-to-end.
Our encoder maps image to the set of latent vectors {z, parameterizing the object-centric light field networks that exist in the scene. We first concatenate fixed pixel coordinates to the image color channels and encode the image into a feature map using a U-net (Ronneberger et al. 2015) encoder. We follow the convention of uORF (Yu et al. 2021) and concatenate constant pixel coordinates without camera information, opposed to how (Stelzner et al. 2021 adds camera position and ray direction channels, to describe the objects in camera space rather than world space, which has shown to generalize more robustly al. 2019). To map the feature grid
Encoder:
to novel viewpoints and objects (TTatarchekko the Slot Attention module (Locatello al. 2020), which
set of latent
vectors describing the objects in the image, achieves this by sampling set of latent vectors, or "slots", from learned slot distribution, and having them dynamically specialize to image entities via recurrent competition to explain parts of the image. The previous work uORF (Yu et al., 2021) leverages the observation that the geometry and appearance of background and foreground elements in images are significantly different and that the model could benefit from disentangling their latent spaces by using scparate slot distribution for cach. This modification of sampling one slot from background slot distribution and the remaining slots from a foreground slot distribution enabled their work to be the first to reconstruct scenes with textured backgrounds of significant. complexity; we adopt this

---

Published in Transactions on Machine Learning Research (05/2023)
architectural change well. The pscudo-code describing the background-aware slot encoding is the same in uORF, but exists in the supplemental material for reference.
Pred.
Ordering Values (Per-Slot)
Min-Subtracted Ordering Values (Per-Slot)
Figure 4: Predicted ordering values (per-slot) for a city-block scene and its minimum-subbraated comparison.
Background Light Field: Following uORF Yu et al. 2021 represent the background via a conditional light field network (r)(r) (r; BBG). (F) is defined in world space, while all foreground scene components hb are defined in camera space. The intuition for this design choice is that since the background geometry is always canonicalized in world space, is casier for the model to learn the background geometry and appearance in world space, whereas since the foreground elements are randomly rigidly transformed in world space, it easier for the foreground components to be queried in camera space rather than have the model perform implicit camera to world space transformation.
Our model encodes an input image and renders novel views I' of the underlying scene at set of query camera positions. We supervise the model with the L2 reconstruction loss [ 21, where Iquery are the ground truth views. We use a deep-feature based perceptual loss (Zhang et al. 2018) on both chair datasets to avoid inherent ambiguities in estimating lighting and geometry at occluded views. Lastly, we impose small penalty ||=||2 on each object regressed code z enforce Gaussian prior.
Losses:
Curriculum Schedule: Empirically, slot encoders are limited in the complexity of scenes that they can disentangle into parts, leading to a prevalance of scenes consisting of simple geometric shapes of uniform color (Eslami et al., 2018 Burgess et al., 2019 Lin et al. 2020). This can by explained by two factors. One is how the difficulty of learning additional components which define factorized slot representation increases with the object complexity and variance. These factorized scene components include the slot distribution parameters to accurately describe the space of objects, decoder weights to reconstruct the factorized objects, and the compositing of the decoded objects into the original domain. And second, there is no principled reason for factorized scene representation to emerge other than compression, i.e., the difficulty of encoding complex scene with a monolithic description, and encourage the model to escape this local minima. We observe that when the state-of-the-art model (Yu et al. 2021) evaluated on a dataset of textured chairs, rather than the uniformly colored chairs they report success on, decomposition fails to emerge as a result of the increased object complexity complicating some combination of the first. two factorizedd-repreenttaale components discussed. This shortcoming is exacerbated in our model, which has fewer constraints on multi-view consistence as 3D-structured representations and has to learn additional parameters of neural compositor, due to the nature of neural light fields (Sitzmann et al. 2021). We observe that our model similarly has high variance in learning factorized representations on a dataset of uniformly colored chairs. However, similar to prior work, our model consistently and reliably learns a factorized representation on the CLEVR-567 dataset, due to the decreased object complexity. We thus propose a curriculum learning strategy, where we use the CLEVR-567 to learn prior of factorized representation before training on more complex scenes. We find that this factorized prior is strong and leads immediate convergence of factorized representation evaluate on (see Sec. [..1 , is to initialize the models for Room- Room-Chair, Room-Diverse, and City-Block with the weights of model that has been trained on the CLEVR-567 datasct. We note that this incremental Icarning curriculum is likely a viable strategy lo advance the scene complexity bound of slot-based aulo-encoders in general. Lastly, we initially render and supervise images at 64 x 64 resolution to efficiently learn the coarse structure and decompasition of scenes, and subsequently supervise at 128 128 to learn more fine object structure. Note however, that we do not need employ higher-resolution supervision via cropped images, the baseline Yu et al. 2021) does, thanks to the efficiency of the light field decoder.
each subsequent dataset evaluation. This strategy, given the datasets we
7

---

Published in Transactions on Machine Learning Research (05/2023)
(a) Decomposition Results
uORF
COLF
uORF
WORF
COLF
Input
View
Novel
View
Novel
Seg.
Input
View
Novel
View
Novel
Seg.
(b) Shadow Reconstruction GT
uORF
COLF
GT
uORF COLF
Figure 5: Qualitative Comparison. In (a), from top to bottom, compare reconstructions of the input view, novel view, well the novel view segementation by the state-of- the-art baseline uORF (Yu et al. (021)) and our method, across CLEVR, chairs-simple, and chairs-diverse. We achieve higher reconstruction quality across each dataset while being orders magnitude more efficient. Interestingly, our method (arguably correctly) assigns shadows to their object slots rather than the background slot and reconstructs them significantly better, highlighted in (b). For video results, please see the supplement.
4 Experiments
We demonstrate that compositional neural light fields, unconstrained by the sampling requirements of volumetric rendering, outperform prior work on unsupervised learning of object-centric 3D) representations while dramatically reducing time and memory complexity. We further demonstrate that object-centric light fields admit scene editing in the from of translation and composing, and allow rendering of scenes with tens of objects at interactive frame-rates. Please find further qualitative results, including video, in the supplemental material.
4.1 Setup
Basclines. On both tasks of scene docomposition and novel view synthesis, we compare to the statc-of-the- art method uORF Yuu el al. 2021]. Since the proposed method and uORF are the only two models in this unsupervised 3D-aware object discovery regime, we also provide a reference point to models without these inductive biases the 2D stato-of-the-art Slot Attention model (no 3D inductive bias) (Locatello al. 2020) on the task of scene decomposition, and a NeRF (Mildenhall al. (0200) autoencoder without any compositionality inductive bias dubbed "NeRF-AE'. Note that the recently proposed GIRAFFF. (Niemeyer

---

Published in Transactions on Machine Learning Research (05/2023)
Table 1: Quantitative Comparison. Our method outperforms state-of-the-art baselines al. 2021 across reconstruction quality metrics, while being orders of magnitude faster and requiring less memory. We also find that COLF often captures shadows where uORF does not.
CLEVR-567
Room-Chair
Sroun-Divesse
Model
LPIPS 4
PSNR
LPIPS
SSIM
PSNR
LPIPS
SSIM
PSNR
NeRF-AF Yu al. [202)
0.1288
0.8658
27.16
0.1166
0.8265
28.13
0.2458
0.6688
24.80
uORF Yu et al 2721
0.0859
0.8971
29.28
0.0821
0.8722
29.60
0.1729
0.7094
25.96
Ouss
0.0608
0.9346
31.81
0.0485
0.8934
30.93
0.1274
0.7308
26.02
Table 2: Quantitative Segmentation metrics. On segmentation metrics of room-scenes, our method performs upproximately on par with uORF while being orders of magnilude faster and achieving belter recoustruction quality. Note that on CLEVR, COLF performs worse because it more accurately reconstructs shadows, assigning those "background' pixels to the foreground; this is reflected in the FG-ART comparison
CLEVR-587
Rocmn Chair
Room Diverse
Model
ARI
NV-ARI
PG-ARI
ARI
NV-ARI f
FG-ARL
ARI 1
NV-ARI
FG-ARI f
Slot Attention
Locatello et al. 2720
3.5 + 0.7
93.2 + 1.5
38.4 + 184
40.2 + 4.5
17.4 11.3
43.8 + 11.7 67.9
nORF Yu 2121
86.3 0.1
83.8 0.3
87.4 0.8
78.8 2.
743 1.9
88.8 4
65.6 4: 1.0
56.9 4 0.2
COLP (ours)
60.0 + 0.4
558 + 0.1
92.4 1.7
85.6 + 0.04
80.7 0.1
89.8 + 0.1
66.5 0.4
52.5 + 0.3
64.7 + 0.7
& Geiger 2020 is an unconditional generative model and can thus not serve as a competitive baseline, as it cannot reconstruct a multi-objoct scene from a given image YYu et al. 2021).
Datascts. We use four datasets. First, to compare our model with uORF on the tasks of scene sogmentation and novel view synthesis, we evaluate models on their proposed three room-scene datasets of increasing scene complexity. Further, to demonstrate compositing of multiple scenes and rendering of resulting scenes with many objects, we also introduce new synthetic dataset of long city block scene with two lanes of car traffic.
The first room-scene dataset proposed by (Yu et al. 2021 is a 3D extension to the CLEVR (Johnson et al.] (017)) dataset. A textureless room is populated with five seven simple geometric shapes (cubes, cylinders, and spheres) and "Rubber" material without specularity. There 1,000 scenes for training and 500 for testing.
CLEVR-6577
The second room-scene dalaset is populated with three to four chairs of the same geometry, and the room features three different floor textures. There are 1,000 scenes for training and 500 for testing. Views of cach scene are captured with a camera at fixed elevation and randomly sampled azimuth, pointed the room center.
Room-Chair:
Room-Diverser The third room-scene dataset is populated with three to four chairs of the geometry sampled from 1,200 ShapeNet chairs (Chang et al. 2015), and the room features fifty different floor textures. There are 5,000 scenes for training and 500 for testing. Views of each scene are captured with a camera at fixed elevation and randomly sampled azimuth, pointed at the room center.
To demonstrate editing and composition of scenes with many objects and large depth range, place instances of four car geometries from the ShapeNet (Chang et al. 2015) dataset two-lane city block. A natural difficulty here is to provide the model sufficient context of all cars placed through the city block al once. To address this, we offer several context views per scene, captured in front of each row of cars placed. At train time, we populate the scene with two rows cars (two context views). The camera is always facing forwards and varies in height from the car height to just above the ground and depth-wise from the beginning to end of the city block. There are 500 scenes for training and we render out one scene for qualitative demonstration.
City-Block:
9

---

Published in Transactions on Machine Learning Research (05/2023)
Implementation Details. On CLEVR-567, we set the background latent vector to 0, since the model needs no information about the unchanging background. This leads to faster model convergence. On City-Block, we discard the background latent and instead use a dedicated background light field network. We pretrain this network on all training images.
4.2 Unsupervised Scene Decomposition
For each test scene in all three room-scene datasets, we encode one of the scene's four captured images as the input view and use the remaining three for evaluation on novel views. We render novel views at the camcra positions the three ground truth query views, and infer segmentation masks for evaluation from the compositor module's s slot masks. Specifically, we map a pixel p in the rendered view to one of the model's 's slots by assigning the slot to which the compositor has yielded the highest contribution weight at p.
Sctup:
Metrics: We use the Adjusted Rand Index (ARI) metric to evaluate our inferred segmentation masks against ground truth segmentation masks. To quantitatively compare with the baseline (Yu et al. 2021). we evaluate with three versions of the ARI: (1) ART on only the input view (2) ART on only the three novel views (ARI-NV), and (3) ARI on only the foreground elements (ARI-FG).
We report quantitative comparisons to the baseline Yuu et al. 2021 in Tbl. and illustrate qualitative comparisons in Fig. ! The results in Tbl. 2 show that our architecture performs well on the task of unsupervised scene decomposition - often outperforming the bascline (Yu et al., 2021 and is competitive when otherwise. The glaring exceptions are the FG-ART and NV-ART scores on the CLEVR-567 dataset, where we benchmark significantly worse. However, as can be seen in the qualitative comparison on the CLEVR-567 scene in Fig. 4 our model performs "worse" when compared to the ground truth segmentation results because our model better reconstructs the shadows of foreground objects than (Yu et al. 2021), which is penalized since the ground truth object masks do not include their shadows. Although some may regard this behavior of assigning shadows to the object inducing them as "incorrect" segmentation, particularly when compared to typical segmentation datasets where shadows manually assigned to the background, note there is no principled way for the model to decide whether shadows are assigned to the background or foreground unless we hand-craft a prior. Especially when considoring that the only training signal novel view synthesis, we argue our model's behavior is more correct, as the rendered shadows are causally related to the objects the object inducing the shadow was removed, the shadow should be removed as well, Thus we want to stress this low score is not evidence that our model cannot form disentangled representations. In fact, we outperform the baseline (Yu et al. 2021 considerably when only considering foreground pixels, confirming that despite this questionable metric, our model indeed forms factorized representations which segment objects in the scene well. Exploring representations with explicit lighting representations, which can naturally factorize shadows and remove this ambiguity, is an exciting future direction too.
Results:
4.3 Novel View Synthesis
Similar to the scene decomposition setup, for each test scene in all three room-scene datasets, we reserve one view as an input view and use the remaining three to evaluate the novel view reconstructions. We render novel views at the query camera positions and compare the reconstructed images with the metrics listed below.
Setup:
Metrics: We evaluate model's novel view reconstructions with the same metrics Yu et al. 2021 : the learned perceptual image patch similarity (LPIPS) metric (Zhang et al. 2018). structural similarity index (SSIM) (Wang et al. 2004). and the peak tignal-to-noise ratio (PSNR).
Results: We report quantitative comparisons Tbl. and qualitative comparisons in Fig. Quantita- tively, we outperform the baseline (Yu et al. 2021 on all metrics despite being orders of magnitude more efficient. Potential contributing factors for their lower-fidelity reconstructions include insufficient latent.
10

---

Published in Transactions on Machine Learning Research (05/2023)
Table 3: Memory Consumption and Performance Comparison. compare the memory consumption and rendering speed of COLF and uORF in rendering single image of encoded scene at 128x 128 resolution for various numbers of slots. Our light field decoder yields over an order of magnitude faster rendering and significantly reduced memory overhead. 1 entries indicate results that exceeded available memory.
2 Slots
Slots
60 Slots
COLF (ours)
uORF
Yu et al.
2021
COLF
uORE
COLF
uORF
Memory Consumption
2.4 GB
7.2 GB
5.0 GB
32.0 GB
31.6 GB
Rendering Speed
166 FPS
6.6 FPS
50.0 FPS
1.4 FPS
6.2 FPS
(a) Scene Editing: Object Manipulation
(b) Scene Editing: Cross-ccene Composition
Context
Object Translation
Source Views
Novel Views of Combined Scene
Scene
Combine, Composite & Render
Scene
Scene
10
Figure 6: Editing and rendering of unbounded scene with many objects. COLF enables compositing and real-time rendering of unbounded scenes with many objects. (a) We edit the scene by translating two cars along the road. (b) We composite ten different scenes reconstructed from views throughout the city block into a single scene with twenty objects.
dimensionality imposed by their memory constraints and coarser volumetric sampling yiclding less detailed reconstruction.
4.4 Rendering Speed and Memory Consumption
Real-time rendering and memory-efficiency are important properties for downstream applications such augmented reality. Thus, compare memory consumption and rendering speed with uORF (Yu et al. 2021 in Tbl. to show how method yields important gains those dimensions. The results highlight that employing a volumetric decoder for object-centric scene representations is considerably expensive and renders any potential downstream applications infeasible even using a modest number of objects pushes the capacity of even large GPUs (seven objects consumes 32GB of memory, e.g.). While uORF YYu et al.) 2021 establishes the direction of combining object-centric representation with 3D-aware representations, an implementation with reasonable support for downstream tasks and applications with larger object sets is important well. Using the light field as our decoder facilitates such applications, requiring only 5GB of memory to render scene with seven objects. Even rendering of scenes with up to 60 objects is feasible without any further optimization.
Memory consumption is often not the only concern, but rendering speed as well. Consider a potential application where the rendering speed is of primary concern, such as interactively manipulating objects in virtual reality: iteratively rendering out 100 frames of scene with four objects at 128x128 resolution takes 1.3 seconds with COLF 31.2 seconds with uORF (Yu al. 2021), making our model uniquely suitable for such applications.
11

---

Published in Transactions on Machine Learning Research (05/2023)
Table 4: Quantitative Comparison Generalization to Larger Object Sets. Our method generalizes well scenes with a larger number of objects than seen during training. We perform comparably to uORF Yu et al. 2021
on the task of spatial decomposition, and outperform them on novel view synthesis.
Model
ARI f
LPIPS +
Slot Attention
5.7
N/A
NeRF-AE (Yu et al. 2021
N/A
0.2201
uORF Yu et al. 2021
83.2
0.1540
COLF (ours)
79.4
0.1128
4.5 Application: Scene Editing and Composition
We demonstrate editing and composition an unbounded scene with many objects. The scone features a city block populated with rows of car models, placed randomly throughout the block's length. The large range in depth from the beginning the end of the block is a challenge for volume rendering- based approaches, which would require setting near and far planes that enclose the full scene, while also sampling along the ray densely enough to guarantee sufficient sampling of objects along the ray. As in Fig. 6 COLF succceds in decomposing the city-block scene into background and foreground car light fields.
Subsequently, we may edit the scene by transforming the object-centric light field networks and placing the editing results. By transforming the input coordinates
additional objects into the scene. for an object-centric light field, we may translate the corresponding object throughout the scene. By collecting object codes across scenes and compositing them together, can generate scene with twenty cars that can still be rendered at interactive framo-rates. Please see the supplemental material for volumetric baseline comparison and further results, including analysis of the cost of attempting to render these scenes at the same fidelity using prior approaches, and discussion on the limitations of using light field representations for scene editing applications.
4.6 Generalization to Cluttered Scenes
We now evaluate our model's ability to generalize to larger number of objects on the packod-CLEVR-11 dataset, which increases the number of objects in the CLEVR-567 dataset from a maximum of seven objects to eleven objects. Qualitative segmentation and view synthesis results are presented in Fig and quantitative results are presented in Table
GT Seg.
COLF
COLF Seg.
GT Seg.
COLF
COLF Seg.
hapur
Input
Novel
Novel
Figure 7: Qualitative segmentation and view synthesis results for scenes with more objects than seen during training, for both input and novel views.
5 Discussion
In summary, we have proposed COLF, novel compositional neural scene representation that parameterizes the 360-degree, 4D light field 3D scene by composing from object-centric neural light fields, thereby enabling the unsupervised discovery of object-centric 3D representations. COLF outperforms previous
12

---

Published in Transactions on Machine Learning Research (05/2023)
state-of-the-art approaches in unsupervised scene decomposition, while being two orders of magnitude faster and significantly less memory-intensive. Several exciting directions for future work remain. While COLF improves over the previous state-of-the-arr in scene decomposition and compositional novel view synthesis, nevertheloss, the scenes we reconstruct are still limited to simple, synthctic scenes. As discussed in the related work, concurrent work improving the performance of the slot. attention inference algorithu offers promising directions for extending our object-light-ficle based representation to more complex 3D scenes. These concurrent works improving the robustness of slot-attention based encoders are rapidly converging real-world scenes, with few approaching real-world driving scenes. Yet, equipped with only 2D-based image decoders, these models struggle to reconstruct scenes well and have no explicit 3D understanding. COLF offers tractable 3D representation to plug-and-play into their architectures. Prior to our contribution, extending these emerging object-centric archilectures with 3D representations via the prior SOTA in 3D object representations Yu et al. 2021 would prove difficult to scale to large worldworld datasets due to the expensive cost of volumo-rendering. COLF's computational advantage over Yu et al. (2021) would be particularly significant in the application of object-centric learners to real-world driving scenes, where the number of cars in given scene be large. With respect to improving COLF's rendering quality, resouuttnnoottionn LFNs may be learned by leveraging neural networks with periodic activation functions (Sitzmann et al. 2020) Fourier Features (Tancik et al.] 2020). Incorporating motion into learning and inference may improve the scene decomposition quality and robustness (Kipf et al. 2021). We believe that such improved inference algorithms for compositional scene representations are the next important step toward applying these models to real-world scenes.
Acknowledgments.
This work is in part supported by Toyota Research Institute (TRI), NSF RI #2211255 AFOSR YIP FA9550-23-1-0127,, Stanford Institute for Human- Centered
ONR MURI Artificial Intelligence (HAI).
References
Edward H Adelson, James R Bergen, et al. The plenoptic function and the elements of early vision, volume 2. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology. 1991.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, et al. Learning physical graph representations from visual scenes. Proc. NewrIPS, 33, 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. or arXio preprinl arXiu:1901 11390, 2019.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shaponet: An information-rich 3d model repository. Xiv preprinl Xrxi 551000301, 2015.
Billy Chen, Daniel Horn, Gernot Ziegler, and Hendrik PA Lensch. Interactive light field editing and compositing. 2006.
Chang Chen, Fei Deng, and Sungjin Ahn. Object-centric representation and rendering of 3d scenes. arXiv preprinl ar Xix::2006.0000 06130, 2020.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proc. CVPR, pp. 5939-52 5948, 2019.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Matthew DuVall, John Flynn, Michael Broxton, and Paul Debevec. Compositing light field video using multiplane images. In ACM SIGGRAPH 2019 Posters, pp. 1-2. 2019.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Cathrin Elich, Martin R. Oswald, Mare Pollefeys, and Jörg Stückler. Semi-supervised learning of multi-object 3d scene representations. arxiv preprint arXiv:2010.04030 2020.
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprint arXiu:206.07766, 2022.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv preprint arXio:1907.13052, 2019.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and GeofTrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprinl arXiv:1603.0855 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Tvo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (206994))1204 1210, 2018.
Fridovich-Keil and Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels:
Radiance fields without neural networks. In CVPR, 2022.
Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14346-14355, 2021.
Steven J Gortler, Radek Grzeszzzuk, Richard Szeliski, and Michael Cohen. The lumigraph. In Proc. SIGGRAPH, 1996.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Jürgen Schmidhuber, and Harri Valpola. Tagger: Deep unsupervised perceptual grouping. arXiv preprint 14:6:660606720 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. arXiv preprint arXin:1708.. 084998 2017.
Klaus Groff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, 2019.
Jindong Jiang, Sepehr Jaughorbani, Gerard de Melo, and Sungjin Ahn. Scalable object-oriented sequential gencrative models. Unknown Journal, 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya. Harada. Neural 3d mesh renderer. In Proc. CVPR, pp. 3907-3916, 2018.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint crXie:2111.122
2021.
Adam R Kosiorck, Hyunjik Kim, Ingmar Posner, and Yee Whye Tch. Sequential attend, infer, repeat: Generative modelling of moving objects. arXiv preprint arXiv: 01 794, 2018
Mare Levoy and Pat Hanrahan. Light field rendering. In Proc. SIGGRAPH, 1996.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. Xixiv preprint arXiv:2001.02400 2020.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction. arXiv preprint
2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiu:2006.15055 2020.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh, Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906. 077511, 2019.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. CVPR, 2019.
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1 14, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1110::111 July 2022. doi: 10.1145/ 3525223.353117.. URL https::/doi..rrr
www.11111//3221.1.. 3530127
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks. arXiv e-prints, pp. arXiv-2103, 2021.
Thu H Nguyen-Phuoe Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. Proc. NewrIPS, 33:6767-677 6778, 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. arXiv preprint ar Ni1:2011.12100, 2020.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assistan Intervention MICCAI 2015, 2015.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for riso-resonution clothed human digitization. In Proc. ICCV, pp. 2304-2311 2314, 2019.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint ar xrXi::2205.10066 14065, 2022.
Vincent Sitzmann, Justus Thics, Felix Heide, Matthias NicBner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. CVPR, 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Proc. NeurIPS 2019, 2019b.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NewrIPS, 2020.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NewrIPS, 2021.
Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the TEEE/CV Conference Computer Vision and Pattern Recognation, pp. 175-184, 2019.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint orXiv:2104.01445 2021.
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Proc. NeurIPS, 2020.
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 3405 3414, 2019.
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d scene representation and rendering. Xrxiv preprinl ar XrX:::2000004 0455,, 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600-61) 612, 2004.
Yiheng Xie, Towaki Takikawa, Shunsuke Sailo, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Proc. Burographics STAR, 2021.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NewrIPS, 2020.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance ficlds from few images. Proc. CVPR, 2020.
Klong-Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arxiv preprint arXfc:2107.07005 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision ond pattern recognition, pp. 586 595, 2018.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplanc images. In SIGGRAPH, 2018
16


Reviews:
[{"review_id": "Ijz6C9SoVP", "reviewer": "Reviewer_Lc2D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The quantitative results in Sec 5.6 of the Suppl. indicate that the proposed method achieves better performance on view synthesis, which partially demonstrates its generalization capability for this task setting. \n\nWeaknesses:\n\n- As the authors mentioned, the results on the scene editing indicate poor generalization of the proposed representation compared to the previous method. This makes the proposed strategy less appealing despite its efficient computation as it would require extra modeling, which may be more costly for complex scenes.\n\n- For the second task in Sec. 5.6, the performance is worse than the uORF in segmentation. While there is ambiguity in whether shadow should be a part of object, a good representation should be able to incorporate object prior to allow the user to remove such artifact. It is unclear how easy to implement this in this framework. \n\n- The presentation of the paper needs some improvement. Compared to uORF, many results did not report the variance and there is some apparent typos (Table 1 in Suppl. COLF 0.794?). Also, the paper would be more clear if some of the results, such as generalization, are placed in the main paper instead of the Suppl.      ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The revised version added two sets of experiments on the generalization performance of the proposed method to the supplement, while the main paper seems largely unchanged. For the new part, the authors provided one example to show the limitation of the light-field representation for the scene editing task, and more results on the task of generalizing to more objects.  \n  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be more convincing if the paper can comment on how much extra effort is needed for achieving similar/better generalization as/than uORF (e.g., on scene editing task). Would the fast speed of this method provide a better strategy in terms of overall cost? \n\n- See weakness 2 and 3 above. ", "broader_impact_concerns": "No statement is provided. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JkebmfkYKN", "reviewer": "Reviewer_UGZL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe paper is very well-written and easy to follow. The paper correctly identifies the scope of the contributions. Furthermore, the contributions seem to be valuable. Related works are adequately covered, in particular, fast nerf approaches. Qualitatively the results seem comparable to uORF; however, given the simplification and the computational boost, I am convinced about the merits of the suggested approach. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the task of unsupervised object discovery in 2D images of a 3D scene. The main idea is to enhance the slot attention model with a light field decoder. The main benefit of light field decoders is computational; however, as they lack the explicit modeling of the underlying view-independent radiance field, composing light fields is challenging. The paper address this challenge with a simple parametrization of light-object order intersection. In turn, they can show competitive results on several object-discover benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No requests ", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tCDtipmJwb", "reviewer": "Reviewer_UksW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I was the reviewer of the past version of this paper. The major issue was 1) the generalization abilities, 2) evaluating scene editing, and 3) results for cluttered scenes. The newly added supplimentary material addressed those concerns and provided further discussions on the usecases of the proposed approach.\n\nThe claims in this paper are supported well, and the limitations are documented. Those requirements meets the quality bar of TMLR, so I'm now leaning towards accepting the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduced a method that uses neural light fields to disentangle objects from the scene in an un-supervised manner. The authors build a parametric, compositive neural light field representation in an efficient manner, and was able to significantly reduce time and memory requirements to discover objects, disentangle objects, and re-render novel views.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have well addressed the comments by the reviewers. 
Decision: Accept (poster)