Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
Authors: Karl Stelzner
Kristian Kersting
Adam R. Kosiorek
Venue: ICLR 2022 Submitted (2022)
Keywords: Representation Learning
Unsupervised Object Discovery
Neural Radiance Fields

Abstract: We present ObSuRF, a method which turns a single image of a scene into a 3D model represented as a set of Neural Radiance Fields (NeRFs), with each NeRF corresponding to a different object. A single forward pass of an encoder network outputs a set of latent vectors describing the objects in the scene. These vectors are used independently to condition a NeRF decoder, defining the geometry and appearance of each object. We make learning more computationally efficient by deriving a novel loss, which allows training NeRFs on RGB-D inputs without explicit ray marching. After confirming that the model performs equal or better than state of the art on three 2D image segmentation benchmarks, we apply it to two multi-object 3D datasets: A multiview version of CLEVR, and a novel dataset in which scenes are populated by ShapeNet models. We find that after training ObSuRF on RGB-D views of training scenes, it is capable of not only recovering the 3D geometry of a scene depicted in a single input image, but also to segment it into objects, despite receiving no supervision in that regard. 

Under review as a conference paper at ICLR 2022
DECOMPOSING 3D SCENES INTO OBJECTS VIA UNSUPERVISED VOLUME SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present ObSuRF, method which turns single image of scene into 3D model represented as a set of Neural Radiance Fields (NeRFs), with each NeRF corresponding different object. A single forward pass of an encoder network outputs a set of latent vectors describing the objects in the scene. These vectors are used independently appearance of each object. We make learning more computationally efficient by deriving a novel loss, which allows training NeRPs on RGB-D inputs without explicit ray marching. After confirming that the model performs equal or better than state of the art on three 2D) image segmentation benchmarks, we apply it to two multi-object 3D datasets: A multiview version of CLEVR, and novel dataset in which scenes are populated by ShapeNet models. We find that after training ObSuRF on RGB-D views of training scenes, it is capable of not only recovering the 3D geometry of a scene depicted in a single input image, but also segment it into objects, despite receiving no supervision in that regard.
condition NeRF decoders, defining the geometry and
INTRODUCTION
The ability to recognize and reason about 3D geometry is key to wide variety of important robotics and Al tasks, such as dynamics modelling, rapid physical inference, robot grasping, or autonomous driving (Battaglia et al., 2013; Chen et al., 2016; Mahler et al., 2019; Driess et al., 2020; Li et al., 2020). While progress has been made on integrating traditional representations of geometry with machine learning systems (Wang et al., 2018; 8; Qi et al., 2017; Nash et al., 2020), they remain difficult to work with due to either undesirable scaling behavior (voxels, point clouds) or discrete structures and inability train on observational data (polygonal meshes).
Recent work has revealed coordinate- neural networks as an alternative. These representations are continuous, independent of resolution scene dimensions, and can directly map 3D coordinates to binary occupancy indicators (Mescheder et al., 2019), signed distances to the shape (Park et al., 2019), or volume radiances (Mildenhall et al., 2020). When used with encoder network, such functions can facilitate learning low-dimensional implicit representations of geometry (Mescheder al., 2019; Kosiorek et al., 2021). They can also render high quality images (Martin-Brualla al., 2021; Barron et al., 2021), and synthesize novel scenes (Guo et al., 2020; Niemeyer & Geiger, 2020).
In this paper, we investigate a different aspect. We ask whether a latent-variable model built around implicit representations may be used to infer semantically meaningful information from given im- age, without human supervision. Such representations may then be used for downstream reasoning tasks. In particular, we focus on obtaining object-basee representations from multi-object scenes in an unsupervised way. Representations factored into objects are beneficial for dynamies mod- elling, visual question answering, and many other tasks (Battaglia et al., 2016; Santoro et al., 2017; Battaglia et al., 2018). Previous works in unsupervised object-based representation learning have mostly focused segmenting 2D images (Eslami et al., 2016; Greff et al., 2020; Locatello et al., 2020). Such methods have remained limited to visually simple, mostly synthetic data, whereas more realistic scenes with complex textures and geometries have remained out of reach (Weis et al., 2020).
To move towards more complex scenarios, we present ObSuRF, a model which learns to decompose scenes consisting of multiple Objects into Superposition of Radiance Fields. We first encode the input image with a slot-based encoder similar to (Locatello et al., 2020), but use the resulting set of latent codes to condition continuous 3D scene functions (Mildenhall et al., 2020; Kosiorek et al.,

---

Under review as a conference paper at ICLR 2022
Iypec Image
Recontractio
Deptls
Segnetiiees
Novel Niew ft
Nevel Viow
Clissip
Figure 1: ObSuRF uses a single image (left) to infer a set of NeRFs (slots) representing different objects. ObSuRF accurately models the geometry and appearance of CLEVR-3D scenes (top), and decomposes them into objects. It also produces details which are not visible in the input image, such the full shape of partially occluded objects, or the shadows on the backsides of objects. This allows rendering the scene from arbitrary angles (right). The MultiShapeNet dataset (bottom) fea- tures a much larger variety of more complicated objects, such that ObSuRF cannot capture of its details. However, it still learns to segment the scene into volumes corresponding to the objects, and reproduce their dimensions, color, and position. For a full demo, please watch the supplementary videos at https: htmpp://ssee //sites. google. com/ com/view/ wtt//viww////////////eeeeeeeeeee obsurf/.
2021) instead of directly generating 2D images. For training the model in 3D, we provide three RGB-D views of each scene, and optimize the model to match the observed depths and colors. To do so, reframe NeRF's volumetric rendering jective, which allows for more efficient training when depth information is available as supervision. After confirming that the resulting model is capable of segmenting 2D images well better as previous approaches, we test on two new 3D benchmarks: A 3D version of CLEVR (Johnson et al., 2017) featuring multiple viewpoints, camera positions, and depth information, and MultiShapeNet, a novel multiobject dataset in which the objects are shapes from the ShapeNet dataset (Chang et al., 2015). We will publish both of these benchmarks and the code for our model with this paper.
Poisson process and derive novel training ob-
RELATED WORK
Learned Implicit Representations of 3D Shapes. Early work focused on predicting the geometry of shapes using occupancy fields (Mescheder et al., 2019) or signed distance functions (SDFs) (Park et al., 2019). These models were trained using ground-truth geometry, for instance by sampling points from the meshes in the ShapeNet dataset (Chang et al., 2015). Mildenhall et al. (2020) subsequently introduced Neural Radiance Fields (NeRF), which predict density and color of points in scene. This admits rendering via differentiable ray tracing through the scene. The required supervision is thereby reduced to small set of posed images. However, ray tracing makes this approach very computationally expensive. Therefore, in this work we opt for a pragmatic middle ground: assume access to views of the training scenes enriched with depth information (RGB-D), and show how this allows for training NeRFs without expensive raymarching. We note that unlike ground-truth meshes, such data may be obtained at scale in the real world with reasonable accuracy using either low-cost camera systems (Horaud et al., 2016) or by estimating depth from stereo vision motion, e.g. (Chang & Chen, 2018; Teed & Deng, 2020).
In its original form, NeRF is fitted to a single scene, which makes too slow for real-time appli- cations, and also ill-suited for representation leariing, the scene is encoded in neural network
2

---

Under review as a conference paper at ICLR 2022
weights. Recent work has used encoded representations of input scenes to condition decoder shared between scenes (Yu et al., 202 la; Trevithick & Yang, 2020; Kosiorek et al., 2021). We take a similar approach, but put greater emphasis on compact and factored representations. Using depth supervision to improve NeRF training has concurrently been proposed by Deng et al. (2021), without however avoiding ray marching and the corresponding performance costs.
Unsupervised Image Segmentation. The work in this area can be broadly grouped in two cate- gories. Patch-based approaches represent scenes as a collection of object bounding boxes (Eslami at, 6016; Kosiorek et al., 2018; Stelzner et al., 2019; Crawford & Pineau, 2019). This encourages spatial consistency in objects, but limits the model's 's ability to explain complicated objects. Scene- mixture models represent scenes pixelwise mixture models, with each mixture component corre- sponding to a single object (Greff et al., 2016; 2017; van Steenkiste et al., 2018; Burgess et al., 2019; Greff al., 2020; Locatello et al., 2020; Engeleke et al., 2020; 2021). Since spatial consistency is not enforced, these models sometimes segment images by color rather than by object (Weis et al., 2020), motivating combinations of the two concepts (Lin et al., 2020). We mitigate these issues by employing NeRF. Explicitly formulated as an MLP of Fourier-encoded spatial coordinates (instead of a CNN, similarly to e.g. Kabra et al. (2021)), 1), it is naturally biased toward spatial coherence. We note that no model in this line of research has been shown to work well on general natural images. Instead they have targeted synthetic benchmarks or constrained real world environments. Some re- cent models use wwwppoont-deeedeee decoders (Nanbo et al., 2020; Chen et al., 2021; Kabra et al., 2021), which allows generating images from novel viewpoints. Since there is nothing to enforce multi-view consistency, these models are unlikely generalize to novel viewpoints.
Unsupervised 3D Segmentation. Most work in this area focuses on segmenting single shapes into parts, and is trained on ground truth geometry. BAE-NET (Chen et al., 2019) reconstructs voxel inputs as a union of occupancy functions. CvxNet (Deng al., 2020) and BSP-Net (Chen et al., 2020) represent shapes as a union of convex polytopes, with the latter capable of segmenting them into meaningful parts. Similarly. UCSG-Net (Kania et al., 2020) combines SDF primitives via boolean logic operations. yielding rigid but interpretable shape descriptions.
Only small number of publications attempt to segment full scenes into objects. Ost et al. (2021) train a model to segment a complex real-world video into objects, but doing so requires a full scene graph, including manually annotated tracking information such as object positions. In GIRAFFE (Niemeyer & Geiger, 2020) object-centric representations emerge when a set of NeRFs conditioned on latent codes is trained as a GAN. However, the focus lies on image synthesis and not infer- ence. Consequently, it is not straightforward to obtain representations for given scene. BlockGAN (Nguyen- Phuoc et al., 2020) achieves similar results, but uses latent-space perspective projections and CNNs instead of NeRFs. RELATE (Ehrhardt et al., 2020) builds on this to allow the manipula- tion of scenes. Elich et al. (2021) present model which encodes a single input image into a set of deep SDFs representing the objects in the scene. In contrast to our work, theirs requires pretraining on ground-truth shapes and operates on simpler scenes without shadows or reflections. Closest to work is system concurrently developed by Yu et al. (2021b), which also learns segment scenes into a set of NeRFs using a slot based encoder. However, they train using NeRF rendering and mitigate its computational cost by adjusting the rendering resolution.
Set Prediction. Objects present in a scene are interchangeable and form set. Any attempt to assign a fixed ordering to the objects (such as left-to-right) will introduce discontinuities as they switch positions (e.g. as they pass behind each other) (Zhang et al., 2020). It is therefore essential to use permutation equivariant prediction methods in such settings. We build on recent works which construct permutation equivariant set prediction architectures (Locatello et al., 2020; Goyal et al., 2021; Kosiorek et al., 2020; Stelzner et al., 2020).
3 METHODS
ObSuRF decomposes 3D scenes into multiple NeRFs, each representing a separate object. We derive principled way of composing multiple NeRFs into single, albeit more expensive (by factor of n)) scene function. To decrease this cost, we show how to make NeRF integration cheaper when ground-truth depth is available at training time.
3

---

Under review as a conference paper at ICLR 2022
Samples along from arbitrary camera
Camera
per slot
Encoder
NeRF Decader
Object-based latent representation
Composc
Render
Render
Input image
Figure 2: ObSuRF architecture. The encoder is given an input image with the corresponding camera pose, and infers object-based latent representation consisting of slots. These slots used in- dependently to condition a shared NeRF decoders. The volumes represented by the resulting scene functions may be rendered individually by querying them along rays coming from arbitrary cam- era. Alternatively, the volumes may be composed in order to render the full scene.
3.1 NEURAL RADIANCE FIELDS (NERFS)
NeRFs (Mildenhall et al., 2020) represent the geometry and appearance of a scene as a neural net- work f (x, d) + (c, mapping world coordinates and viewing direction d to a color value с and density value a. To guarantee coherent geometry, the architecture chosen such that the density is independent of the viewing direction d. To simplify notation, refer to the individual outputs of f(-) as (xx) and c(x, d). NeRFs allow rendering images using classic ray marching techniques for volume rendering (Blinn, 1982; Kajiya & von Herzen, 1984). Specifically, the color C(r) corresponding the ray is given by
C(r) C(r)= - [ T())(((())(((()).... d)dt,
with transmittance T(1) = exp (r(('))dt' Since this procedure is differentiable, NeRFs are trained by minimizing the L2 loss between the rendered colors Ö(r) and the colors of the training image C(r), i.c. (reR((T) [((r) Crr) This is expensive, however, approximating Eq. (I) requires evaluating í many times per pixel (256 in Mildenhall et al. (2020)). We will show that, if depth is available, we can instead train NeRFs with just two evaluations per pixel, reducing the required computation by factor of 128 per i
3.2 VOLUME RENDERING AS POISSON PROCESS
While ray marching in Eq. (1) allows computing the ray color, we are also interested in finding the distribution of distances that give rise this color -tis will allow using depths RGB-D data for supervision. To this end, we show that Eq. (1) derives from an inhomogenous spatial Poisson process (Blinn, 1982; Møller Waagepetersen, 2003).
Consider ray r(t) Xo d/ traveling through a camera at position Xo along direction d. The probability that light originating at point r(t) will not seatter and reach the camera unim- peded (transmittance) is equal to the probability that events occur in the spatial Poisson process T(1) exp fu (r(('))d('). NeRF does not model light sources explicitly, but assumes that lighting conditions are expressed by the color value of each point. As a result, the light emitted at the point r(t) in the direction the camera -d has the color c(r(t), d), and intensity is propor- tional to the density of particles (r(t)) at that point. Consequently, the amount of light reaching the camera from r(t) proportional
p(1) = (r())T(()
TThis reduction affects only the decoder side of autoencoder model.
4

---

Under review as a conference paper at ICLR 2022
In fact, under mild assumptions, p(t) is exactly equal to the distribution of possible depths t at which the observed colors originated". We provide the full derivation in Appendix B.1.
We can now reframe Eq. (1) as the expected color value under the depth distribution p(t):
(r) = Bpp (ooep))((((())) (c(((()))))))))))))) - p(()c(((()) d)dt.
Typically, one chooses a maximum render distance (far as the upper bound for the integration interval. This leaves the probability p(t ffar - (( that light from beyond lür is missed. To account for that, Eq. (3) is renormalized by dividing by T (twrr after approximating it with samples 0 S t, S tjar-
3.3 FAST TRAINING ON DEPTH AND COLORS
We now turn to the optimization objectives we use to train ObSuRF. Even though Mildenhall et al. (2020) use hierarchical sampling scheme, training NeRFs requires many function evaluations for each ray and is therefore extremely expensive. Moreover, as we show in Appendix B.2, this sam- pling produces a biased estimate for C(r), and, consequently LNeRF- This is caused by the nested integration in Eq. (I): If too few samples are collected, there is significant chance that thin, high- density volumes are missed entirely, even though they would dominate the color term C(r) if it were to be evaluated analytically (see e.g. Rainforth et al. (2018) for a thorough treatment of such issues).
To avoid this computational cost during training, we use depth supervision by training on RGB-D data, i.e. images for which the distance between the camera and the visible surfaces is known. Instead of integrating over t, this allows us to directly maximize the depth log-likelihood log p(t) in Eq. (2) for the known values of t. While we still need to approximate the inner integral, an unbiased estimate can be obtained with uniform random samples from q(-) = Uniform (0, 4rr)) as log p()) = log (r(()) (gt) [o(r(())]] Since (rr(()) is likely to be large near the surface t and close to zero for 1' < 1, the variance of this estimator can be very especially at the beginning of training. To reduce the variance, we importance sample 1' from proposal q'(-) with higher density near the end of the integration range,
log = log (r((t) - Er) (f(dd))))
(4)
In practice, we set (-) to be an even mixture of the uniform distributions from 0 to 0.98t and from 0.98/ to i.e., we take 50% of the samples from the last 2% of the ray's length.
We fit the geometry of the scene by maximizing the depth log- ligliikeldoood of Eq. (4). Staying with our probabilistic view, we frame color fitting as maximizing the log-likelihood under a Gaussian distribution. Namely, since know the point r(t) at which the incoming light originated, we can evaluate the color likelihood as PCC r((), d) = NCC | c(r(t), d).a)) with fixed standard devia- tion de. Overall, evaluating the joint lok-liiel p(t, C)) requires only two NeRF evaluations: At the surface rtt) and at a point r(t') between the camera and the surface. In practice we take the surface sample r(t+e e) with e Uniform(O, 8). This encourages the model to learn volumes with at least depth ö instead of extremely thin, hard to render surfaces.
3.4 COMPOSING NERFS
We are interested in segmenting scenes into objects by representing each of them as an independent NeRF A scene is then represented by set of NeRFs f1...., fu, each with densities d; and colors C, We now show how to compose these NeRFs into a single scene function. We arrive at the same equations
(Niemeyer& & Geiger, 2020; Guo et al., 2020; Ost et al., 2021), but derive them by treating the composition of NeRFs as the superposition (Møller & Waagepetersen, 2003) of independent Poisson point processes, yielding probabilistic interpretation of the algorithm.
Specifically, we assume that, while we move along the ray r, each of the n Poisson processes has chance of triggering an event independently. The total accumulated transmittance T(t) from r(t)
2 Note that the transmittance is symmetric, and that both occlusion and radiance depend on the density a (-). Therefore, if a light source is present at xo. the distribution of points along it illuminates is also given by Fq. (2). It can be convenient to think of ray marching as shooting rays of light from the camera into the scene instead of the other way around.
5

---

Under review as a conference paper at ICLR 2022
to Xo the probability of encountering events along the way should therefore be equal to the product of the transmittances (t) of each process:
T(t) = I Ti(t) exp (-/-" (-1²2 (o (r((tt)) )
(5)
2m
This is equivalent another Poisson process with density (xx) ,x))
To compute the color value, we additionally need to determine to what degree each of the component NeRFs is responsible for the incoming light. Following Eq. (2), the probability that NeRF is responsible for the light reaching the camera from depth t is p(t, i) (r(()))T()). Similarly to Eq. (3), we compute the pixel color by marginalizing over both depth & and component i, yielding
((r) EE,.ppl (c.
Σ p(t, ()),)r()
can also be useful to marginalize only one of the two variables. Marginalizing yields the depth distribution p(t) = 2, i(t, which we use to render scenes via hierarchical sampling as in (Mildenball et al., 2020). By marginalizing i, one obtains a categorical distribution over the com- ponents p(t)= = = p(t, i), which we use to draw segmentation masks. Finally, in order to compute the color loss derived above, we use the expected color c(x, d) = Σ, cx (x/(())/((x)).
3.5 LEARNING TO ENCODE SCENES AS SETS OF NERFS
We now deseribe the full encoder-decoder architecture of ObSuRF (see Figure 2), which can infer scene representations from a single view of that fore-for any scene from a large dataset. ObSuRF then turns that representation into a set NeRIs, which allows rendering views from arbitrary viewpoints and provides full volumetric segmentation of the scene.
The encoder network fene infers set of latent codes Fn (called slots) from the input image and the associated camera pose. Each slot represents a separate object, the background, or is left empty. By using a slot 2, to condition an instance of the decoder network face. we obtain the component NeRF fi(-.)) face i 24). In practice, we set the number of slots n to one plus the maximum number of objects per scene a given dataset to account for the background.
Encoder. Our encoder combines recent ideas set prediction (Locatello et al., 2020; Kosiorek et al., 2020). We concatenate the pixels of the input image with the camera position Xo. and a positional encoding (Mildenhall et al., 2020) of the direction d of the corresponding rays. This is encoded into a (potentially smaller) feature map y. We initialize the slots 2, by sampling from a Gaussian distribution with learnable parameters. Following Locatello et al. (2020), we apply a few iterations of cross-attentio between the slots and the elements of the feature map. interleaved with self-attention between the slots (similar to Kosiorek et al. (2020)). These two phases allow the slots to take responsibility for explaining parts of the input, and facilitate better coordination between the slots, respectively. The number of iterations is fixed, and all parameters are shared between iterations. The resulting slots form the latent representation of the scene. We note that it would be straightforward to use multiple views as input to the encoder, but we have not tried that in this work. For further details, we refer the reader to Appendix C.I.
Decoder. The decoder largely follows the MLP architecture of Mildenhall et al. (2020): We pass the Fourier-encoded inputs x through a series of fully-connected layers, eventually outputting the density a (x) and hidden code h. To condition the MI.P latent code 2;. we the code to shift and scale the activations at each hidden layer, a technique which resembles AIN of (Dumoulin et al., 2017; Brock et al., 2019). The color value c(x, d) is predicted using two additional layers from h, d, and pixel in the feature map y selected by projecting the query point x onto it. This conditioning follows (Yu et al., 2021a), except that we only use it to predict color, in order to not compromise the idea of the slots encoding the segmented geometry. We explored using to predict an explicit linear coordinate transform between the object and the world space as (Niemeyer & Geiger, 2020; Elich et al., 2021), but have not found this to be beneficial for performance. We also note that we deliberately choose to decode each slot 24 independently. This means we can treat objects as independent volumes, and in particular between slots during decoding using e.g. attention (as proposed by Kosiorek et al. (2021)) would likely improve the model's ability to output complex visual features such as reflections, but would give up these benefits Further details are described in Appendix C.2.
can render one object at time. Computing interactions

---

Under review as a conference paper at ICLR 2022
Table 1: Average foreground ARI on 2D datasets (in %, mean + standard deviation across 5 runs, the higher the better), compared with values reported in the literature (Locatello et al., 2020; Greff al., 2020; Burgess al., 2019;
Model
Sprites (bin)
Sprites
CLEVR
ObSuRF
74.4 + 1.8
92.4 + 1.3
98.3 0.8
sel. runs
74.4 + 1.8
93.1 + 0.3
99.0 + 0.0
SlotAtt.
69.4 + 0.9
91.3
98.8 0.3
IODINE
64.8 17.2
76.7 5.6
98.8 0.0
MONet
9.8
0.6
van Steenkiste al., 2018), Best val- ues are bold.
R-NEM
68.5 + 1.7
(*) For one run on Sprites and runs on CLEVR, ObSuRF learns to segment background from foreground. While desirable, this slightly reduces the foreground ARI scores extracting exact object outlines is more difficult. To quantify this, we also report results with these runs excluded.
Training. The model uses minibatch of scenes at every training iteration. For every scene, we use one image (and its associated camera pose, but without depth) as input to the encoder. This yields the latent state {= We then average the loss on random subset of rays sampled from the available RGB-D views of that scene. Finally, we average across the minibatch.
One advantage of 3D representations is that they allow us to explicitly express the prior knowl- edge that objects should not overlap. This is not possible in 2D, where one cannot distinguish between occluding and intersecting objects. We enforce this prior by adding the overlap loss Co(r) = Σ, o, (r(t) max; o, (r(f),, optimizing the overall loss L = log log C*) + (r)) Experimentally, we find that Lo can prevent the model from learning object geometry at all when present from the beginning. We therefore start with ko 0 and slowly increase its value in the initial phases of training. In turn, we find that we do not need the learning rate warm-up of Locatello al. (2020). We describe all hyperparameters used in Appendix E.
EXPERIMENTAL EVALUATION
ObSuRF is designed for unsupervised volumetric segmentation of multi-object 3D scenes. As there are no published baselines for this setting, we start by evaluating it on 2D images instead of 3D) scenes. This allows us gauge how inductive biases present in a NeRF-based decoder affect un- supervised segmentation. We also check how a slotted representation affects reconstruction quality (compared monolithic baseline). We then move on to the 3D setting, where we showcase our model on our novel 3D segmentation benchmarks; we also compare the reconstruction quality to the closest available baseline, NeRF- VAE (Kosiorek et al., 2021).
Metrics. We evaluate segmentation quality in 2D image space by comparing produced segmenta- tions to ground-truth using the Adjusted Rand Index (ARI, Rand (1971); Hubert & Arabie (1985)). The ARI measures clustering similarity and is normalized such that random segmentations result in a score of O, while perfect segmentations in a score 1. In line with Locatello et al. (2020), we not only evaluate the full ARI, but also the ARI computed on the foreground pixels (Fg-ARI; according to the ground-truth) Note that achieving high Fg-ARI scores is much easier when the model is not attempting segment the background, i.c. to also get a high ARI score: This is because ignor- ing the background allows the model to segment the objects using rough outlines instead of sharp masks. To measure the visual quality of our models' reconstructions, we report the mean squared error (MSE) between the rendered images and the corresponding test images. To test the quality of learned geometry in the 3D setting we also measure the MSE between the depths obtained via NeRF rendering and the ground-truth depths in the foreground (Fg- (ig-Depth-MSE). We exclude the back- ground as a concession the NeRF-VAL baseline, as estimating the distance of the background in the kind of data we use can be difficult from visual cues alone.
4.1 UNSUPERVISED 2D OBJECT SEGMENTATION
We evaluate our model on three unsupervised image segmentation benchmarks from the multi-object datasets repository (Kabra et al., 2019): CLEVR, Multi-dSprites, and binarized Multi- -SpriedS We compare against four recent models as statc-of-the-an baselines: SlotAttention (Locatello et al., 2020), IODINE (Greff al., 2020), MONet (Burgess et al., 2019), and R-NEM (van Steenkiste
7

---

Under review as a conference paper at ICLR 2022
at. 2018). We match the experimental protocol established by Locatello et al. (2020): On dSprites, we train on the first 60k samples, on CLEVR, we select the first 70k scenes for training, and filter out all scenes with more than 6 objects. Following prior work, test on the first 320 scenes of each validation set (Locatello et al., 2020; Greff et al., 2020) and process CLEVR images by center- cropping and resizing to 64 x 64.
To adapt our encoder to the 2D case, we apply the positional encoding introduced by Locatello et al. (2020), instead of providing camera position and ray directions. To reconstruct 2D images, we query the decoder on fixed grid of 2D points, and again drop the conditioning on viewing direction. During training, we add small amount of Gaussian noise this grid to avoid overfitting to its exact position. Similar to previous methods (Locatello et al., 2020; Greff et al., 2020), we combine the 2D images which the decoder produces for each slot by interpreting its density outputs o, as the weights of a mixture, i.e. we use the formula Ö(r) = -,,, c(()).//. As we show in Appendix B.3, this is equivalent to shooting ray through a superposition of volumes with constant densities o; and infinite depth. We train the model by optimizing the color reconstruction loss.
In Table 1, we compare the foreground ARI scores achieved by our model with those from the literature. Visualizations are given in Appendix A.I. We find that our model performs significantly better on the Sprites data, especially the challenging binary variant, indicating that our decoder architecture imparts useful spatial priors. On CLEVR, our model largely matches the already nearly perfect results obtained by prior models. We note that like (Locatello et al., 2020), our model occasionally learns to segment the background in a separate slot, which is generally desirable but slightly reduces the Fg-ARI score as discussed above. Additionally, as we report in A.2, we have found ObSuRF to produce better reconstructions than an ablation with a monolithic latent code instead of slot based representation, confirming that the latter is also helpful for prediction quality.
Training ObSuRF on these 2D datasets takes about 34 hours on a single V100 GPU. This is roughly times faster than SlotAttention (133 hours on one V100 (Locatello et al., 2020)) and almost 40 times more efficient than IODINE (One week on 8 V100s (Greff et al., 2020)). We attribute the efficiency gains compared to SlotAttention to our decoder architecture, which processes each pixel individually via small MLP instead of using convolutional layers.
4.2 UNSUPERVISED 3D OBJECT SEGMENTATION
To test ObSuRF's capability of learning volumetric object segmentations in 3D, we assemble two novel benchmarks. First, we adapt the CLEVR dataset (Johnson et al., 2017) from (Kabra et al., 2019) by rendering each scene from three viewpoints, and also collecting depth values. The initial viewpoint is the same as in the original dataset, the others are obtained by rotating the camera by 1209/2409 around the z-axis. As in the 2D case, we restrict ourselves to scenes with at most objects in them. Second, we construct MultiShapeNet, a much more challenging dataset which is structurally similar to CLEVR. However, instead of simple geometric shapes, each scene is popu- lated by 2-4 objects from the ShapeNetV2 (Chang et al., 2015) 3D model dataset. The resulting MultiShapeNet dataset contains 11 733 unique shapes. We describe both datasets in detail in Ap- pendix D. In contrast to the 2D case, we utilize the full 320 x 240 resolution for both datasets. Due to our decoder architecture, we do not need to explicitly generate full size images during training, which makes moving to higher resolution more feasible.
When testing our model, we provide single RGB view of a scene from the validation set to the encoder. We can then render the scene from arbitrary viewpoints by estimating ((r) (Eq. (6)) via hierarchical sampling as in (Mildenhall et al., 2020). We set the maximum length thr of each ray to 40 or the point at which it intersects 2 = - 0.1, whichever is smaller. This way, we avoid querying the model underneath the ground plane, where has not been trained and where we cannot expect sensible output. We use rendering obtained from the viewpoint of the input to compute the reconstruction MSE. In order to obtain depthmaps and compute the depth-MSE, we use the available samples to estimate the expected value of the depth distribution Epco [t] = =f' t p(!)dt. We draw segmentation masks by computing arg max p()). Finally, we also render individual slots by applying the same techniques to the individual NeRFs, following Eqs. (2) and (3). In order to
highlight which space they do and do not occupy. we use the probability p(t S trx) as an alpha channel, and draw the slots in front of checkerboard background.
8

---

Under review as a conference paper at ICLR 2022
CLEVR-3D
MultiShapeNet
Model
C-MSE
D-MSE
Fg-ARI
ARI
C-MSE
D-MSE
Fg-ARI
ARI
ObSuRF
0.31
0.07
96.9
92.0
0.60
1.28
94.7
61.1
without Lo
0.60
0.25
94.0
1.4
0.82
3.32
94.9
16.9
w/o px. cond.
0.78
0.10
95.7
94.6
1.81
3.44
81.4
64.1
w/o px. cond., Lo
0.80
0.12
85.5
4.83
1.78
3,40
94.4
16.5
w/ NeRF loss
0.69
0.25
97.8
18.0
2.08
4.00
58.75
3.1
w/ D-NeRF loss
1.11
0.15
97.9
15.7
4.7
8.95
12.8
0.2
NeRF-VAE
4.7
1.03
5.5
110.3
Table 2: Quantitative results the 3D datasets. C-MSE scores colour MSE x x10 D-MSE
depth MSE values computed on the foreground only. ARI scores are percent, best values in bold. Lo is the overlap loss.
We report results for the full ObSuRF model, which uses both the overlap loss Lo and pixel condi- tioning, and a number of ablations. These do not use the overlap loss (w/o Lo), or pixel conditioning (w/o pixel cond.). Additionally, we report results for ObSuRF trained with vanilla NeRF rendering (w/ NeRF loss) and with NeRF rendering plus depth loss of (Deng et al., 2021) (w/ D-NeRF loss). Details on these ablations are given in Appendix C.5. The slot sizes are 128 for CLEVR-3D and 256 for MultiShapeNet. As baseline, an autoencoder where the NeRF decoder conditioned on a latent extracted from a single input image. However, it trained without depth information, and does segment the scene. Instead, maximizes the evidence lower-bound (ELBO), comprised of a reconstruction term and a regular- izing KL-divergence term. It also uses much larger latent representations of size 8 x 128, and an attention based decoder. While the differences in architecture make training times impossible directly compare, we estimate that due to our RGB-D based loss, ObSuRF required 24x fewer evaluations of its scene function than NeRF- over the course of training.
compare NeRF-VAE which, similarly to model,
Figure and Table 2 contain qualitative and quantitative results, respectively. We find that ObSuRF learns to segment the scenes into objects, and to accurately reconstruct their position, dimensions, and color. On the MultiShapeNer dataset, the large variety among objects causes the model to sometimes output vague volumes instead of sharp geometry, whereas on CLEVR-3D, it stays close to the true scene. We observe that similarly to the 2D models, our model does not learn to segment the background into its slot without Lo. With this additional loss term however, it learns to also segment the background, leading to much higher ARI scores. As expected, pixel conditioning increases the accuracy of predicted colors and geometry. On CLEVR3D, both versions of NeRF training yields similar results to our loss, whereas on MultiShapeNet, they fail to produce accurate segmentations or geometry, with depth supervision Deng et al. (2021) actually performing worse. In this more challenging setting, we also find that our training method converges much faster, as shown by the learning curves in Appendix A.3. Compared NeRF-VAF we find that our model achieves much lower reconstruction errors on both datasets.
5 CONCLUSION
We present ObSuRF, a model that segments 3D scenes into objects represented NeRFs. We have shown that learns infer object positions, dimensions, and appearance on two challenging and novel 3D modelling benchmarks. Importantly, it does using purely observational data, without requiring supervision on the identity, geometry or position of individual objects.
Since ObSuRF learns to infer compact object representations that have to be well informed of their geometry (due to the model structure), using these representations may be beneficial for downstream tasks including robot grasping, dynamics modelling, visual question answering. A key step in this direction is to improve inference capabilities and robustness to noisy depth supervision, which will enable working with real-world data.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation engine of physical scene understanding. Proceedings of the National Academy of Sciences (PNAS), 110(45), 2013. 1
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Inter- action networks for learning about objects, relations and physics. In NeurIPS, 2016.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzaler Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. In arXiv: 1806.80066.111 2018.
James F. Blinn. Light reflection functions for simulation of clouds and dusty surfaces. In SIG- GRAPH, 1982. 4, 14, 15
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019. 6
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. In arXiv: 1901. /1390, 2019. 3,7
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Ried 3D Model Repository. In arXiv: 1 2015. 2, 8, 19
Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In CVPR, 2018. 2
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d
scenes. In arXiv:2006.061 2021. 3
Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monoc- ular 3d object detection for autonomous driving. In CVPR, 2016.
Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. Bae-net: Branched autoencoder for shape co-segmentation. In ICCV, 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary
space partitioning. In CVPR, 2020. 3
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In AAAI, 2019. 3
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. CVPR, 2020.
Kangle Deng, Andrew Liu, Jun-Ya Yan Zhu, and Deva Ramanan. Depth Depth-supervised nerf: Fewer views and faster training for free. In arXiv:2107.02791 2021. 3, 19
Danny Driess, Jung-Su Ha, and Marc Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. In RSS, 2020.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017. 6
Sébastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy J. Mitra, and Andrea Vedaldi. Relate: Physically plausible multi mujti-object scene synthesis using structured latent spaces. In NewrlPS, 2020.
10

---

Under review as a conference paper at ICLR 2022
Cathrin Elich, Martin R. Oswald, Marc Pollefeys. and Joerg Stueckler. Semi-supervised learning of multi-object 3d scene representations. In arXiv:2010.040330, 2021. 3,
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020. 3
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-V2. Inferring Unordered Object Representations without Iterative Refinement. arXiv preprint arXiv:2104.09958, 2021.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with gen- erative models. In NeurIPS, 2016. 1,3
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In ICLR, 2021. 3
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber.
Tagger: Deep unsupervised perceptual grouping. In NeurlPS, 2016. 3
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber, Neural expectation maximization. In NeurlPS, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICMI., 2020. 3, 8
Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. In arXiv:20/2.08503, 2020. 1,5
Radu Horaud, Miles Hansard, Georgios Evangelidis, and Menier Clément. An overview of depth cameras and range seanners based on time-of-flight technologies. Machine Vision and Applica- fions, 27, 2016.
I. Hubert and P. Arabie. Comparing partitions. Journal Classification, 1985. 7
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 2
Rishabh Kabra, Chris Burgess, Loie Matthey, Raphael Lopez Kaufman, Klaus Greff, Mal- colm Reynolds, and Alexander Lerchner. Multi-object datasets. https: /ttpt/ggttt///////////// com/ 2019. 7, 8,
deepmind/mut mwtiaaddmmuiiiiiiimeiiiemiiiiilt
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P Burgess. SIMONe: object representations via unsupervised video decomposition. In Neural Information Processing Systems, 2021. 3
temporally-abstracte
James Kajiya and Brian von Herzen. Ray tracing volume densities. In SIGGRAPH, 1984. 4, 15
Kacper Kania, Maciej Zięba, and Tomasz Kajdanowicz. Ucsg-net unsupervised discovering of constructive solid geometry tree. In NewrlPS, 2020. 3
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In NewrIPS, 2018. 3
Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. Conditional set generation with transform- ers. In arXiv:20006 16841, 2020. 3,6 6
Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo J. Rezende. Nerf-vae: A geometry aware 3d scene generative model. In DxXiv:/04400088 2021. 1, 3, 6, 7, 13, 16, 18
Yikai Li, Jiayuan Mao, Xiuming Zhang, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Perspective plane program induction from single image. In CVPR, 2020.
11

---

Under review as a conference paper at ICLR 2022
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In ICLR, 2020. 3
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. In NeurlPS, 2020. 1, 6, 7, 8, 13, 17, 18
Jeffrey Mahler, Matthew Matl, Vishal Satish, Michael Danielezuk, Bill DeRose, Stephen McKinley, and Ken Goldberg. Learning ambidextrous robot grasping policies. Science Robotics, 4(26),
2019. 1
Ricardo Martin Bartin-Bruaall,, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019. 1,2
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 1,2, 4, 5, 6, 8, 16, 17, 18, 19
Jesper Møller and Rasmus Plenge Waagepetersen. Statistical inference and simulation for spatial point processes. CRC Press, 2003. 4, 5, 14
Li Nanbo, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi- object scenes from multiple views. In NeurlPS, 2020. 3
Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. In ICMI,, 2020.
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. In NeurIPS, 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In urXiv:2011.121000 2020. 1, 5,
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In arXiv: 2011.10379, 2021. 3,5
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 1.2
Thomas Porter and Tom Duff. Compositing digital images. In SIGGRAPH, 1984. 16
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
Tom Rainforth, Robert Cornish, H. Yang, and A. Warrington. On nesting monte carlo estimators. In ICMI, 2018. 5
William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the Amer- ican Statistical Association, 1971.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In NeurlPS, 2017.
Karl Stelzner, Robert Peharz, and Kristian Kersting, Faster attend-iiffer-eepaat with tractable proba- bilistic models. In ICML, 2019.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Generative adversarial set transformers. In Workshop Object-Oriented Learning at ICML, 2020.
12

---

Under review as a conference paper at ICLR 2022
Model
Sprites (bin)
Sprites
CLEVR
Table 3: Testset reconstruction MSEs
x x10* of our model compared to the NeRF-AE ablation, which uses only single latent vector.
ObSuRF
1.34 + 0.15
0.64 + 0.11
0.41 + 0.06
NeRF-AE
4.82
2.60
1.75
Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. In ICLR, 2020.
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d scene representation and rendering. In arXiv:2010.04595, 2020.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In ICLR, 2018. 3,7
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. ECCV, 2018.
Marissa Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker. Unmasking the inductive biases of unsupervised object rep- resentations for video sequences. In arXiv:2006. 200,00000600344 2020. 1,3
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields
from one or few images. CVPR, 2021a. 3, 6, 17
Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In arXiv:2107.07905, 2021b. 3
Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Espool: Learning set representations with featurewise sort pooling. In 2020, 3
2020.
A ADDITIONAL RESULTS
A.1 2D VISUALIZATIONS
In Fig. 3, we provide visualizations of ObSuRF's 2D results.
A.2 2D ABLATION
To quantify the impact of learning representations which are segmented into slots, we construct baseline which uses only a single slot, called NeRF-AE. To keep the total computational capacity comparable to our model, we set the size of this single latent vector to be equal to the total size of the slots ObSuRF is using. We also double the size of the hidden layers in encoder and decoder. Still, as we show in Table 3, our model consistently achieves much lower reconstruction errors. This confirms previous results indicating that representing multi-object scenes using set of permutation equivariant elements is not just helpful for segmentation, but also for prediction quality (Kosiorek al., 2021; Locatello al., 2020).
A.3 RUNTIME COMPARISON
We show learning curves of ObSuRF (without Lo) and the NeRF based ablations in Figure 4. While the models behave comparably on CLEVR3D, ObSuRF converges much faster and to a much better segmentation result Shapenet. Training time the z-axis is measured in scene function evaluations, as these make up at least 99% of the computational cost training each model.
B PROOFS AND DERIVATIONS
We start by providing detailed derivations for the results referenced in the main text.
13

---

Under review as a conference paper at ICLR 2022
Figure 3: Visualisations of ObSuRF's output given samples from the validation set of Sprites (top)
and CLEVR2D (bottom).
B. DERIVATION OF THE NERF DEPTH DISTRIBUTION
In Section 3.1, we noted that the distribution p(t) shown in Eq. (2) arises both as the distribution of points along r who contribute to the color observed illuminated by a light source located at Xg. Here, we start by showing the latter.
=O0), and as the distribution of points
B.1.1 DISTRIBUTION OF LIGHT ORIGINATING AT
Following the low albedo approximation due to Blinn (1982), ignore indirect lighting. In order for light originating at Xg to illuminate r(t) directly, it must travel along the ray without any scatter- ing events. The rate of scattering events is given by the inhomogenous spatial Poisson process with finite, non-negative density (x) (Møller & Waagepetersen, 2003). We are therefore looking for the distribution over the position of the first event encountered while traveling along r, also called the arrival distribution of the process (Møller & Waagepetersen, 2003). In a Poisson process, the probability of encountering no events along a given trajectory r(!) with 2 (transmittance)
T(t)= exp (f(-((((t))att
The probability that light scatters before moving past some point to is therefore p(t S tv) = T(to). Let assume that lim T(to) = 0. i.e. each ray of light encounters scattering event eventually. This is for instance guaranteed if the scene has solid background. In that case, T(t) is the cumulative distribution function for p()). since it is non-decreasing, continuous, and
14

---

Under review as a conference paper at ICLR 2022
1.0
1.0
0.4
0.4
NeRF Loss
0.2
0.2
OESuRF wdo px cond.
D.NeRF Loss
0.0
0.5
1.0
1.5
2,0
2.5
3.0
3.5
4,0
4,5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Function
1c10
Evaluations
1e10
0.0200
0.012
NeRF Loss
ORSuRE
0.0150
0.0125
CSE
0.006
0.0100
0.0075
0.004
0.0000
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Scene Function Evalustions
1e10
Scene Function Evaluations
1610
Figure 4: Learning curves of ObSuRF and its ablations measuring Fg-ARI (top) and C-MSE (bot- tom) when training on CLEVR3D (left) and Multishapenet (right). For better comparability with NeRF-based training, all ObSuRF models are trained without the overlap loss Co.
fulfills 7(0) 0, ((xo) 1. We therefore recover the density function p(t) by differentiating:
d p(() (1 T(())
(8)
di
= -T(() di L'- (r(((())) =o(r()))(())
(9)
(10)
B.1.2 DISTRIBUTION COLORS OBSERVED AT Xo
As discussed in Section 3.1, NeRF does model light sources explicitly, but assumes that lighting conditions are expressed by each point's RGB color value. The observed color C(r) at point xo along ray r is then blend of the color values along r(t) (Blinn, 1982; Kajiya & von Herzen, 1984). The intensity with which we can observe the color emitted from point r(t) direction -d of the camera is proportional to the density (r()).. Let & denote the constant of proportionality. Since T(t) is the probability that light reaches Xo from rtt), the we can observe the color c(r(t), d) at xo with intensity ko (t)))
order to obtain the distribution p(t) over the points from which the colors we observe Xo originate, must normalize this term dividing by the total intensity along ray p(t)
ko (T()))((
(r(()((())
(11)
Jo (rrr
To ((r(t)))((()d'
As we have shown in Eq. (10), the antiderivative of (r(t)) is -T(t). The value of the improper integral is therefore
[
lim (+00
(((((t)))(()))) lim T(0) T(t) 1.
(12)
t+000
15

---

Under review as a conference paper at ICLR 2022
Consequently, pt) (rt)))T((). This procedure may be viewed continuous version of alpha compositing (Porter & Duff, 1984).
B.2 ESTIMATING NERF IS BIASED
As counterexample, consider the following situation in which a ray passes through thin but dense white volume, before hitting black background. For simplicity, we use grayscale color values.
100
50 S 51
(r((t)
10
if 80 otherwise,
(13)
if 50 < 1 S 51 otherwise.
c(r(t).d)
(14)
We then have T50) 1, but T(51) (800) exp(-100), Since T(+) is the CDF of (Eq. (10)),
this means almost all of the incoming light is coming from the thin volume at t E [50, 51]:
p(50 < 1. S 51) - exp(-1000
(15)
p(t 80) exp(-100).
(16)
Using Eq. (3), we find that the color value is almost exactly one:
((r = - exxx(-1100) ~ 1.
(17)
Let now consider the result of estimating C(r) using h 50 stratified samples f1, .1.... tx. with far = 100. In this case, one sample 4 is drawn uniformly from each of k evenly sized bins:
(²==) Far
4 ~ Uniform
Far
(18)
Only one of the bins, the one for 125 Uniform(50, 52). covers the volume at [50, 51]. However, since this volume only makes up half the range of this bin, we have p(50 < l28 < 51) = 1/2. Therefore, 50% of the time, sampling in this way will result samples which miss the volume at [50, 51] entirely. In those cases, we have c(r(4)). d) O for all i. As a result, the estimated color value is (r)) O. Even if the estimated color is 1 in all other cases, this sampling scheme will be a biased estimator for the true color values:
E., [ot)(n]] S ~
((r).
(19)
Collecting second round of samples based the estimated densities as proposed by Mildenhall et al. (2020) does not change the result. In the 50% of cases where the thin volume was missed entirely during by the first set of samples, we have (r((t)) piecewise-constant PDF used for sampling the second round of samples will equal O for 50 < t < 51, and none of the newly collected samples will cover the relevant range. Therefore, Eq. (19) also holds when the samples i; were collected using hierarchical sampling.
for all S 80. As result, the
We note that this effect may be somewhat mitigated by the fact that NeRF models typically use different scene functions for coarse and fine grained sampling (Mildenhall et al., 2020; Kosiorek et al., 2021). The coarse scene function may then learn to model wider volumes which are easier to discover while sampling, instead of the exact scene geometry. However, it seems clear that this will not allow decreasing the number of samples arbitrarily, c.g. to 2 as we did this paper.
B.3 SUPERPOSITION OF CONSTANT DENSITIES YIELDS A MIXTURE MODEL.
Here, we illustrate how a mixture model, like the one we use in Section 1. naturally arises when superposition of volumes with constant densities a; and colors is rendered. Following Eq. (5) and writing Σ, os the transmittance along a ray r is
T(t) = exp (-f² odd' exp(-at).
(20)
16

---

Under review as a conference paper at ICLR 2022
As in Section 3.4, we can then write
p(t, (T(())) og exp(-ot).
(21)
Integrating over to obtain the distribution over the volumes yields
p(i) l p(l,
oi - exp( - ol)
(22)
Using (x(x = Ct2 and following Eq. (6), we find that the observed color will then be
Bamp()) [c]]
(23)
which is the desired mixture model.
C ARCHITECTURES
We now present the architectures used in our experiments. When we refer some of the dimensions in terms of a variable, we specify the concrete values used in Appendix E. ObSuRF uses exclusively ReLU activations.
C.I ENCODER ARCHITECTURE
Due to the differences in resolution and complexity, used different feature extractor architectures for the 2D and 3D experiments. The slot attention module was identical in both cases.
Feature Extractor for 2D Datasets. Following Locatello et al. (2020), we use a convolutional network with layers to encode the input image for the 2D datasets. Each layer has a kernel size of 5 x 5, and padding of 2, keeping the spatial dimensions of the feature maps constant at 64 x 64. Each layer outputs a feature map with d,, channels. After the last layer, we apply the spatial encoding introduced by Locatello et al. (2020). Again following Locatello et al. (2020), process each pixel in the resulting feature map individually using a layer normalization step, followed by two fully connected (FC) layers with output sizes dn, dz).
Feature Extractor for 3D Datasets. For the 3D experiments, we adapt the ResNet- 18 architecture to achieve higher model capacity. We start by appending the camera position and positional encoding (Mildenhall et al., 2020) of the viewing direction to each pixel of the input image, and increase the input size of the ResNet accordingly. We remove all downsampling steps from the ResNet except for the first and the third, and keep the number of output channels of each layer constant dh. Given an input image of size 240 x 320, the extracted feature map therefore has spatial dimensions 60 x 80, and dz channels. We apply the same pixelwise steps described for the 2D case (layer normalization followed by two FC layers) obtain the final feature map with d, channels.
Slot Attention. We apply slot attention described by Locatello al. (2020) with slots for m iterations, except for the following change. At each iteration, after the GRU layer, we insert a multi- head self attention step with 4 heads to facilitate better coordination between the slots. Specifically, we update the current value of the slots via
= MultiHead(z, z).
(24)
After m iterations, the values of the slots form our latent representation 21,... 2n. We apply the same number of iterations m during both training and testing.
C.2 DECODER ARCHITECTURE
Here, we describe how density and colors are predicted from the spatial coordinates x and a condi- tioning vector If pixel conditioning is not used, then is simply equal to of the latent codes 2;. Otherwise, we concatenate 2; with the feature map pixel d=c where r, c are the image space coordinates of the query point x projected onto the feature map d,, following Yu et al. (2021a).
To then predict color and density, we first encode the x via positional encoding (Mildenhall et al., 2020), using ny frequencies, with the lowest equal to 2 m. We process the encoded positions
17

---

Under review as a conference paper at ICLR 2022
Parameter
Description
2D Data CLEVR-3D
MultiShapeNNe
d₂
Dimensionality of slots
64
128
256
da
Dimensionality of hidden layers
64
128
256
m
Number Slot Attention iterations
5
nf
Number frequencies for pos. enc.
16
16
ky
Exponent for lowest frequency Maximum density
-5
-5
max
10
10
de
Standard deviation of color dist.
0.2
0.2
8
Noise added to depths
0.07
0.07
ko
Maximum overlap loss coefficient Start of Lo ramp up End of Lo ramp up
0.05
0.03
20000
20000
40000
40000
Batch Size Rays per instance
128
64
64
4096
2048
Table 4: Hyperparameters used for the experiments with ObSuRF.
using a layer MI.P with hidden size d,- After each hidden layer, we scale and shift the current hidden vector h elementwise based on z. Specifically, at each layer, we use learned linear map to predict scale and shift parameters a, B (each of size d2,) from z, and update h via == (h + () a.
In the 2D case, we output values at the last layer, namely the log density log and the RGB-coded colors c. In the 3D case, we output one value for the density, and d, values to condition the colors. We append the view direction d to the latter, and predict the actual color output using two additional FC layers of size [d,.3], 3), followed by a sigmoid activation. In the 3D case, we find that is necessary to set a maximum density value, as the model can otherwise arbitrarily inflate the depth likelihood p(t) by increasing the density in regions which are known to always be opaque, such as the ground plane. We do this by applying sigmoid activation to the decoder's density output, and multiplying the result by a max. obtaining values in the range [0, mma)].
C.3 NERF-AE BASELINE DETAILS
For the NeRF-AE baseline (Table 3), we adapt encoder architecture output single vector instead of feature map. To this end, we first append a positional encoding (Mildenhall et al,, 2020) of each pixel's coordinates to the input image. This replaces the spatial encoding described above, since the method described by Locatello et al. (2020) requires a full-sized feature map to be applicable. We double the number of channels to 2d,,, but set the convolutional layers to stride 2, obtaining an output with spatial dimensions 4 x 4. As above, each of these vectors is processed via layer normalization and two fully connected layers, this time with output sizes [2dr, 2d,]. We flatten the result to obtain a vector of size 32 da, and obtain the final vector using an MLP of sizes [4d" 2d',, d' This replaces the slot attention component described below. To keep the total dimensionality of the latent space identical, we set d' ndz.
C.4 NERF-VAE BASELINE DETAILS
We compare ObSuRF NeRF- on the quality of image reconstruction and depth estimation. We use NeRF-VAE with the attentive scene function and without iterative inference as reported by Kosiorek al. (2021), with the majority of parameters the same. The only differences stem from different image resolution used in the current work (240 x 320 as opposed to 64 x 64 in (Kosiorek et al., 2021)). To accommodate bigger images, we use larger encoder with more downsampling steps. Specifically, we use [(3, 2), (3, 2), (3, 2), (3, 2) (4, 1)] block groups as opposed to [(3, 2), 3, 2). (4, 1)], which results in an additional 4x downsampling and additional residual blocks. We used the same parameters to train the NeRF-VAE on both datasets: Models were trained for 2.5 x 10a iterations with batch size 192, learning rate of x 10 and Adam optimizer. The training objective to maximize the ELBO, which involves reconstructing an image with volumetrie rendering. We sample 32 points from the coarse scene net, and we reuse those points plus sample additional 64 points for evaluating the fine scene net. Kosiorek al. (2021) were able to set the integration cutoff
18

---

Under review as a conference paper at ICLR 2022
far 15, but for the data generated for this paper, this value too small and used fla = 40. This effectively reduces the number of points sampled per unit interval, which might explain higher reconstruction and depth-estimation errors.
C.5 NERF TRAINING ABLATION DETAILS
The ablations involving training via NeRF-style ray marching largely use the same architecture as ObSuRF, except for the following changes:
No pixel conditioning is used, as we have found that this causes the model to fail to seg- ment, likely because it allows the model achieve good loss scores without relying on the slots.
No type of overlap loss is used.
The maximum density not limited to maximum value, instead it is predicted using a simple ReLU activation.
Following Mildenhall al. (2020), learn two instances of the decoder for the coarse and fine rendering passes, respectively.
For training on CLEVR3D, we collect 48 samples each for the coarse and fine rendering pass. For shapenet, we use 64 samples each. To account for the increased memory cost, we reduce the batch size to 16, and the number of rays per instance to 256 and 192, respectively.
For depth supervision, MSE loss was placed the estimated depths. Following Deng et al. (2021), this loss was weighted with the factor À = 0.1 for CLEVR3D. For Mul- tiShapeNet, tried 0.1 and 0.04, but found that neither causes the model to properly segment the scene. The results above were obtained using À = 0.04.
D DATASETS
Here, we describe both of the 3D datasets used.
D.1 CLEVR-3D
To allow for maximum interoperability with the 2D version of CLEVR, we use the scene metadata for the CLEVR dataset provided by Kabra al. (2019). While this data does not specify the original (randomized) camera positions, we were able to recover them based on the provided object coordi- nates in camera space via numerical root finding. We were therefore able exactly reconstruct the scenes in Blender, except for the light positions, which were rerandomized. To ensure coherent background in all directions, we added copy of the backdrop used by CLEVR, facing in the oppo- site direction. We rendered RGB-D views of the scene from the original camera positions and two additional positions obtained by rotating the camera by /240 around the axis. We test on the first 500 scenes of the validation set, using RGB images from the original view points as input.
D.2 MULTISHAPENET
For the MultiShapeNet dataset, start with the same camera, lighting, and background setup which was used for CLEVR-3D. For each scene, we first choose the number of objects uniformly between 2 and We then insert objects one by one. For each one, we first uniformly select one of the chair, table, or cabinet categories. We then uniformly choose shape out of the training set of shapes provided for that category the ShapeNetV2 dataset (Chang et al., 2015), leaving the possibility for future evaluations with unseen shapes. We insert the object at a random in [-2.9, 2.9 scale its size by a factor 2.9, and choose its = position such that the bottom of the object is aligned with occlusions, we compute the radius If a previously inserted object has radius r', and their zyydisttace d < 1.1(r + r'), we consider the objects to be too close, remove the last inserted object, and sample a new one. If the desired number of objects has not been reached after 20 iterations, we remove all objects and start over.
position
0. To prevent intersecting objects and reduce the probability of major
the newly inserted object's 's bounding circle in the zly-paane.
19

---

Under review as a conference paper at ICLR 2022
E MODEL PARAMETERS
We report the hyperparameters used for ObSuRF in Table 4. In addition to the architectural param- eters introduced above, we the importance of the standard deviation of the color distribution ac for tuning the relative importance of color compared to depth. We also report how we ramp up the overlap loss Lo: From the beginning of training the iteration at which we start the ramp up, we set ko = 0. During the ramp up period, we linearly increase ko until it reaches the maximum value ko. We train using the Adam optimizer with default parameters, and an initial learning rate of 4e 4. We reduce the learning rate by a factor of 0.5 every 100k iterations. Finally, we note that for the 3D models, we used gradient norm clipping during training, i.c. at each iteration, we determine the L2 norm of the gradients of our model as if they would form a single vector. this norm exceeds
divide all gradients by that norm. When training MultiShapeNet, we find that very rare, extremely strong gradients can derail training. After the first 5000 iterations, we therefore skip all training steps with greater than 200 entirely.
20


Reviews:
[{"review_id": "o0oaROTRLV0", "reviewer": "Reviewer_8UAh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, is pleasant to read and the formulation seems sound. The authors clearly justify their decisions and overall their work provide valuable insight about the problems of competing approaches in the literature. The experimental validation seems well designed and the results out-perform the state of the art for both 2D and 3D datasets.\n\nRegarding the novelty of the work, there are two main aspects: First, the unsupervised decomposition of the scene, which is somehow incremental, given that it's achieved by applying the slot-based approach of [Locatello et al. 2020] to NeRFs, and also very related to [Yu et al. 2021]. Second, the training speed-up using depth training data, which seems to be the most significant contribution. However, this was also explored in a recent work by Kangle Deng et al. \"Depth-supervised NeRF: Fewer Views and Faster Training for Free\". \n\nI do understand that since this work was published in arxiv in July 2021, according to ICLR guidelines the authors are not required to compare their work against it but given that the most significant contribution is so similar, I would find necessary to reference the paper and briefly mention it in the related work. \n\nThe authors comment on the fact that it's possible that some of the reconstruction error gains are due to the additional (depth) supervision. This renders the comparison a bit unfair given that the present work is using additional training data. I miss further discussion, in a similar fashion as the ablation experiment with the overlap loss, but focusing on the depth training data. \n", "summary_of_the_paper": "The authors propose a method to synthesize novel views of 3D scenes while inferring a decomposition of the scene into multiple objects.  The compositional reasoning is achieved by a slot-based encoder so that there's no need for specific supervision on the object categories. The authors also contribute with a novel loss function that exploits depth training data to improve the sampling strategy and therefore increasing training speed as well as lowering the reconstruction error. The results out-perform the state-of-the art, and are validated against 2D and 3D datasets.", "summary_of_the_review": "I would recommend this paper for acceptance as long as no other reviewers have major concerns. The paper conveys a good contribution. It's quite unfortunate that those two recent works (Kangle Deng and Yu et al. , both in July 2021) clash significantly with the two main novelties of this work, but as stated above, those papers are too close to the submission deadline to be considered double submission. In any case, I find the citation of the work of Kangle Deng. et al. a requirement for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FIoB3wSaDuw", "reviewer": "Reviewer_1keq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the idea to use more information allows to use a more efficient algorithm. The topic is out of my field yet the derivations seem theoretically sound. The experimental evaluations (using simulated data) demonstrate that the algorithm works, at least on very simple data.\n\nI have a few questions and remarks:\n\n- What does \"Eq (3) is renormalized\" mean? Please explain that procedure.\n\n- \"a significant chance that thin, high-density volumes are missed\": Standard hardware for capturing RGB-D data likely does not capture thin, high-density volumes either, thus the proposed approach would not be successful for such data. This somewhat implicit argument (the proposed approach can solve thin, high-density volumes) should be removed, thus. \n\n- Can the authors drop a few words about the derivation of log p?\n\n- Eq (4): Why are 50% of the samples taken from the last 2% of the ray? What happens is the ray is sampled differently?\n\n- maximizing Eq 4 under a Gaussian distribution: Is this assumption justified? Can it be violated? If so, which attributes would such a part of a volume have?\n\n- How is the joint log-likelihood p(t,C) defined? Why are only 2 evaluations of the NeRF necessary?\n\n- What do the authors mean by \"we use the code to shift and scale the activations at each hidden layer\"?\n\n- By Fig 1, I assume that the number of objects is provided for each image? Stated a bit differently, it seems that differently many slots are used for the input images? That reduces the claim in the title about the unsupervised segmentation since providing the number of objects can be considered as a weak supervision signal.\n\n", "summary_of_the_paper": "This paper is about unsupervised segmentation both of 3D volumes and of images. The idea is to learn a set of Neural Rendering Fields (NeRFs) which explain a given image. To reduce computationally effort, the authors assume RGB-D data and show how that reduces the computational burden by two magnitudes. The proposed algorithm can learn segmentation masks and a set of NeRFs that generate each individual object and the background.", "summary_of_the_review": "A major disadvantage of using NeRFs is their need for very expensive GPUs. I like that the authors address this issue and propose a solution. The price that one has to pay for that (RGBD instead of RGB, providing the number of objects) can be justified, often. Furthermore, I suppose that this paper will trigger follow-up works which improve the weaknesses of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2ntJssUwByE", "reviewer": "Reviewer_VrrK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- Decomposing a scene into objects in 3D is a challenging problem and solving this could potentially be useful for the area of explainable AI. \n\n- The probabilistic interpretation of ray marching in Section 3.2 is interesting and somewhat novel in the context of NeRF, but this piece of knowledge is well understood in the physically based rendering community.  \n\nWeakness:\n- The problem authors seek to address is not defined clearly. The title and the\nmain concept of the paper are inconsistent. According to the title, a new 3D scene segmentation method is proposed. However, the input of the pipeline is 2D images which is prone to be 2D instance segmentation.\n- Lack of novelty. \nUsing depth to facilitate training is proposed[1].\n[1]Depth-supervised NeRF: Fewer Views and Faster Training for Free\nComposing NeRFs has been explored before, as acknowledged by the authors in the citations. This paper arrives at similar formulations but from the probabilistic point of view. \n\n- The presentation sometimes gets complicated with too long background information (Section 3.1, 3.2, 3.3) that causes some confusions. ", "summary_of_the_paper": "The paper presents an array of techniques for solving the problem of decomposing a single image of a scene into a set of NeRFs, each of which represents an object in the scene. The decomposition requires accurate estimation of scene depth and the segmentation of the scene into objects. To achieve this, the authors propose an enhanced version of NeRF that operates on RGB-D images where its rendering only requires two evaluations per each camera ray. The authors show that the proposed method works effectively on CLEVR-3D and a custom built dataset named MultiShapeNet. \n", "summary_of_the_review": "Summary\nThe authors use an encoder network to infer a set of slots for each object contained in the input image and then these features are used in Nerf to generate novel views. However, it lacks the motivation of leveraging nerf lack. The authors must make it clear. In addition, it lacks experiments to verify the proposed methods. Evaluation of advanced 3D object dataset such as Google Scanned Objects is necessary. Overall, the paper should be rejected.\n\nQuestions to authors\n\n1) The paper seems to include some contributions that do not directly solve the problem. For example, can the decomposition work with traditional NeRF and ray marching instead of the proposed ray evaluation? \n\n2) Section 3.3 mentions that the ray marching in NeRF is biased, which is true and a well known phenomenon in volume rendering. \nThen the authors claimed that the proposed ray evaluation is unbiased. Could you provide a formal proof? \n\n3) The comparison does not include results that evaluates segmentation and depth separately using state-of-the-art methods for each aspect. How does this method fare compared to those SOTAs? \n\n4) Ablation study is lacking. Overall I can see that the proposed method outperforms a few baselines but there is not much analysis into the proposed method. Statistics about running time should be shown too as the proposed ray evaluation could be faster than traditional ray marching in NeRF. \nFor contribution in Section 3.3., output quality compared to NeRF should also be shown. \n\n5) The experiments are done with synthetic datasets only. Since the proposed method is based on NeRF, how well does it work with real photographs? \n\nDetailed comments\n\n- Abstract: the motivation of segmentation is not clear. Infering a set of NeRFs, one for each object, from a single input image is a good approach, but the connection to segmentation of the scene is left unexplained. \n\n- The derivations are sometimes quite lengthy and sometimes the motivation is not so clear. For example, Section 3.1, 3.2, 3.3 only provides background, and then Section 3.4 (which appears quite late in the text) actually tackles the problem. \n\n- The experiments are only reported with quantitative results. Having some visualizations of the segmentation and depth and some novel view renderings in the results would be nice. \n\n- When comparing segmentation results, please add IoU metric. \n\n\nRating\n\nThe paper sounds weak in the sense that some of its contributions are on improving NeRFs while the decomposition part is rather marginal. The presentation is sometimes misleading. There are a few concerns about the experiments and baselines as well. Therefore, my current rating is below the acceptance threshold. I invite the authors to try considering the questions as commented. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2i8dtyHN6t", "reviewer": "Reviewer_bAmB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper a bit hard to follow, as it heavily relies on previous works. However, I think successfully builds on previous works and shows promising results on the used synthetic data.\n\nStrengths:\n* Using depth to reduce the number of samples considerably is an intuitive way to reduce the computational complexity of training NeRFs.\n* The proposed method considerably outperforms (quantitatively) the previous methods under comparison in Tables 1 and 2.\n\nWeaknesses:\n* The segmentation part seems to be already proposed in  \"Object-centric learning with slot attention (Neurips2020)\". What is the difference?\n* The conclusion section could be improved to be more specific on the main paper achievements/observations.\n* Visual comparison against the previous methods in Table 1 and/or 2 are missing.\n* No results on real-world data are provided.\n* Similarly, the results seem to be only good on the CLEVR dataset which contains very simple geometries. For the ShapeNet dataset, the results seem to be rather blurred.", "summary_of_the_paper": "The authors presented a method to infer a neural radiance field per object in a scene from a single input view. \nuses  Object-centric learning with slot attention\n\nThe proposed architecture takes the shape of an autoencoder. The encoder side takes a single image and pose as inputs and generates N latent codes for each object in the scene following  Object-centric learning with slot attention (NeurIPS2020). The latent codes condition shared NeRF decoders to define the geometry and appearance of each object. The full scene is reconstructed by superposing the multiple Radiance Fields.\n\nFurthermore, to reduce the computational complexity of NeRFs, the authors propose to use GT depth maps to avoid full ray marching.", "summary_of_the_review": "The results shown in this work seem promising and advance towards single view 3D scene understanding. However, due to the mentioned weaknesses in the main review, I am leaning towards a \"marginally below the acceptance threshold\" score.\n\nUpdate:\n\nThe authors have addressed the comments in my initial review and showed improved results on the Shapenet dataset. Therefore I am updating my recommendation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops a method for decomposing 3D scenes into objects by coupling NeRF decoders to representations produced by a slot-based encoder.  After the discussion phase, reviewer ratings are mixed with three on either side of above/below threshold, and one higher (but low confidence) accept score.

Drawbacks include limited novelty, as stated by Reviewer 8UAh: "the unsupervised decomposition of the scene, which is somehow incremental, given that it's achieved by applying the slot-based approach of [Locatello et al. 2020] to NeRFs".  Reviewer bAmB likewise mentions this issue.  Reviewer VrrK: "some of its contributions are on improving NeRFs while the decomposition part is rather marginal". Reviewer VrrK also raises concerns about lack of experiments on real data: "Since the proposed method is based on NeRF, how well does it work with real photographs?"

The AC agrees with the marginal rating of the reviewers and is particularly concerned with overall novelty of the proposed pipeline and question of applicability beyond simulated data.  More work seems required to solidify an experimental case on real images.
Decision: Reject