INFERNO: Inferring Object-Centric 3D Scene Representations without Supervision
Authors: Lluis Castrejon
Nicolas Ballas
Aaron Courville
Venue: ICLR 2022 Submitted (2022)
Keywords: object discovery
scene representation
object-centric representations
3D rendering

Abstract: We propose INFERNO, a method to infer object-centric representations of visual scenes without relying on annotations. Our method learns to decompose a scene into multiple objects, each object having a structured representation that disentangles its shape, appearance and 3D pose. To impose this structure we rely on recent advances in neural 3D rendering. Each object representation defines a localized neural radiance field that is used to generate 2D views of the scene through a differentiable rendering process. Our model is subsequently trained by minimizing a reconstruction loss between inputs and corresponding rendered scenes. We empirically show that INFERNO discovers objects in a scene without supervision. We also validate the interpretability of the learned representations by manipulating inferred scenes and showing the corresponding effect in the rendered output. Finally, we demonstrate the usefulness of our 3D object representations in a visual reasoning task using the CATER dataset.

Under review as a conference paper at ICLR 2022
INFERNO: INFERRING OBJECT-CENTER 3D SCENE REPRESENTATIONS WITHOUT SUPERVISION
Anonymous authors
Paper under double-blind review
ABSTRACT
We propose INFERNO, method to infer object-centric representations of visual scenes without relying on annotations. Our method learns to decompose a scene into multiple objects, with each object having a structured representation that dis- entangles its shape, appearance and 3D pose. To impose this structure we rely on recent advances neural 3D rendering. Each object representation defines lo- calized neural radiance field that is used to generate 2D views of the scene through a differentiable rendering process. Our model is subsequently trained by minimiz- ing reconstruction loss between inputs and corresponding rendered scenes. We empirically show that INFERNO discovers objects in scene without supervision. We also validate the interpretability of the learned representations by manipulat- ing inferred scenes and showing the corresponding effect the rendered output. Finally, demonstrate the usefulness of our 3D object representations in visual reasoning task using the CATER dataset.
INTRODUCTION
Inferring objects and their 3D geometry in scene is a fundamental ability of biological visual systems (Kahneman et al. 1992, Roelfsema et al. 1998 Spelke et al. 1993). Replicating this ability in machine is promising step towards visual reasoning valuable several applications involving object manipulation, navigation or forecasting.
Recent works QJang al., 2019 Locatello al., 2020 Burgess et al. 2019) have shown that neural networks can learn object-centric representations from low-level perceptual features. They learn to recognize the objects a visual scene from singe image without relying on supervision. However, most of those approaches only consider the 2D structure of images and ignore the underlying 3D geometry of the visual scenes. On the other hand, Neural Radiance Fields (NeRFs) (Mildenhall al. 2020) have demonstrated that differentiable renderers can be combined with gradient-based optimization to learn high-fidelity 3D scene reconstructions. NeRFs have been subsequently used to learn 3D-aware generative models, including compositional scene models (Niemeyer & Geiger 2021).
In this work, leverage these recent advances in object-centric representation learning (Locatello jet al., (020) and 3D modelling through implicit functions (Mildenhall et al.] 2020 Niemeyer & Geiger [021] and propose INFERNO, model which infers a structured representation of objects and their 3D poses from single image. Each object is represented by latent variables characterizing its shape and appearance, together with an explicit representation of their 3D poses (translation, scale and rotation). The object representations are then decoded using implicit functions that are localized the scene according to the objects poses and combined together generate 2D output view. Our model does not need supervision and instead is fitted through minimizing reconstruction loss, akin to an auto-encoder.
Disentangling the object appearance and pose in a scene representation allows for the model to manipulate a visual scene. In particular, we demonstrate that INFERNO learns interpretable object poses, which we can modify and render to alter the pose of object in a scene. We also validate that our approach learns meaningful representations for object discovery and visual reasoning. More specifically, show that approach obtains competitive performance discovery benchmark (Johnson et al. 2017 Greff et al.) 2019) as well as for the snitch localization visual reasoning task of CATER (Girdhar & Ramanan 2019).
the CLEVR6 object

---

Under review as a conference paper at ICLR 2022
In summary contributions the following:
We propose model able infer and render 3D representations composed of multiple objects, each of them modeled by an implicit function and explicitly localized in the scenc.
We show that the representations learned by the model are interpretable and amenable to manipulations.
We demonstrate that the inferred representations useful for downstream tasks by show- ing competitive performance in object discovery and reasoning tasks.
2
METHOD
Cananical Objects
Transformed Objects
Slots
Representation
Shape
3
Aрpeоrance
Pase
Input
Appearance
Resolution Rendering
Reconstruction
Shaps
Pase
Rackground
Shaps
INFERENCE
RENDERING
Figure 1: Model Overview: We propose INFERNO, model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer structured 3D representation. The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects transformed and located in the overall scene according to their pose. We combine objects and background and render a low-resolution scene given camera location. The input reconstructed by upscaling the low resolution scene.
We propose INFERNO (Infer NeRF Objects). The goal of our method to infer object-centric 3D) scene representations from single 2D views. Given image x E T5. Hxxx x3 learn an inference function fe that maps images to scene representations s = fz( = (0). o2. OK, Obg c). Scenes are composed K objects o,, a background object obg and camera location
Each object is composed of three tensors o = (o, (ophape o, oPP, ooP,o (ose). The object shape shape Pahape and object appearance oP Daaape are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function. The object location o.ose [24x4 is an affine matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.
The background object 04 = shape bg aPP) only models 3D shape and color, and its location is fixed, encompassing the back-of-scent eube. We also define camera matrix E 3x4 that defines the location of the scene camera and determines a 2D projection of the 3D scene.
To optimize inference function formulate optimization problem in which minimize a reconstruction loss over dataset, similar to an auto-encoder. We define rendering function 9-, that takes input scene representation and generates 2D view of that scene & (f(()x)) We
2

---

Under review as a conference paper at ICLR 2022
assume isotropic Gaussian likelihood model with unit covariance and optimize the probability of the data under our model, which is equivalent to minimizing the mean squared error of inputs and reconstructions:
0.00 arg p(X) arg min
Σ (a.- - (x([=((((()))))
7,8
7,9
In the following sections we describe more detail our inference mechanism, our rendering pipeline and their implementation.
2.1 RENDERING PIPELINE
We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al. 2020) with a similar setup as that of GIRAFFE (Niemeyer& & Geiger 202T). NeRF a function 9r that defines a 3D shape implicitly. It takes as input 3D location (Ir y, 2) and 2D viewing direction d (1, Φ) and outputs an occupancy value a and a color value = (r, g. b). NeRFs are usually implemented using fully connected neural networks. Additionally, the inputs usually embedded into higher- dimensional space using positional encodings that embed locations and viewing directions into higher dimensional spaces R. and R Pa respectively.
9+ R R.
R
(d)) - (a,a)
To represent multiple 3D shapes with the same NeRF function, we can augment it with latent vari- ables that determine which shape is being modeled (Schwarz et al. (020). NeRFs are usually augmented with two random variables: one random variable μ E Tahnp defines the shape of the entity being modeled, while v Ro models its appearance. In practice, this specialization is enforced by making the occupancy output a function of only the shape latent, while the color output is conditioned on the appearance latent.
of : x Pa x IR Dshape x XR Ropo + + R3
(7(), - (a, a)
(3)
In INFERNO, we share single parametrization of NeRF function across all objects. Each ob- ject specific shape and appearance defined by the shape and appearance latent variables, which correspond to the object attributes ohhape o The background is defined as another NeRF with separate parameters. The background NeRF also has shape and appearance latent variables to model different backgrounds.
The pose of an object o; in the scene is determined by the affine transformation matrix dese. We denote the coordinate system of the NeRF function of an object the object space, and the coordi- nate system of the scene (and the background NeRF) as scene space. Given an object pose, we can convert points from the scene space the object space by applying the oose transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix.
To render a scene, we cast rays from each pixel in the 2D plane defined by a given camera to the 3D scene. We evaluate NeRFs different points along given ray, and integrate their occupancy and color outputs to determine pixel values. Rays might traverse multiple object NeRFs in addition the background NeRF, To determine the occupancy and color of points described by multiple NeRFs, we first query each NeRF at those particular points. To query the object NeRFs, we first need transform the points from scene space the particular object space. Then, we compose the results of each NeRF with pooling function C, which in our case is weighted average:
N
N
(a
aaa;
(4)
1=1
Since rendering with NeRFs as originally proposed computationally expensive, we render output views fixed low resolution. Low resolution scenes are then upscaled to the desired output

---

Under review as a conference paper at ICLR 2022
resolution using convolutional neural network, keeping the entire rendering pipeline differentiable. Additionally, low resolution scenes are rendered with additional channels, allowing for more detailed upscalings beyond those possible when just rendering low resolution RGB outputs. For more details about NeRF rendering and compositional NeRF models refer to (Mildenhall et al. 2020 Niemeyer & Geiger 2021).
2.2 INFERENCE
Given the rendering pipeline. the goal of our method is to infer representations that reconstruct given scene. Our inference mechanism computes image features through a neural network encoder and then extracts K object and background slots. These slots are then mapped to our structured scene representation through learned neural networks.
To extract image features for each object and background slots, we use Slot Attention (Locatello et al., (020)). Slot Attention is mechanism that maps set of K entities, called slots, to image features without annotations. It extracts image features H from a given input using a resolution-pressee preserving convolutional encoder. These features are then attended to by set of K randomly sampled slots т ~ Ni, or) of dimension D, where μ and a are learnable parameters. We denote by the matrix concatenating all the sampled slots. Slots attend spatial chunks of the input features through soft-attention u TQ"", where T = k(T) and Q = q(T) ) are the embeddings of the inputs and slots respectively. The attention weights are normalized through softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region w so, Umarr(u)) The weighted average of I according to the attention weights is then computed and fed to a GRU network, to update each slot value: Ty GRU (u, I, TT) Vj K. Multiple rounds of soft attention are performed iteratively refine the slots. For more details about Slot- Attentinn, please refer to (Locatello et al. (2020).
Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-conneed individual objects. More concretely, we map object slots to their 3D pose in the scene through 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin) et al. 2018
networks that operate on
RELATED WORK
3.1 3D SHAPE REPRESENTATIONS
There are different ways to represent 3D geometry such as voxels meshes (Rematas & Ferrari 2020 Gkioxari et al, 2019). For example, the GAN models of Nguyen-Puuoc et al. (2019) (2020) successfully use voxel- based representations to render images. Voxel-based methods have trouble scaling up to high resolutions as the size of a voxel representation scales cubically with the resolu- tion.
Recently, the use of functions that implicitly model 3D volumes has gained popularity (Park et al.
2019
Mescheder al. 2019 Sitzmann al., 2019 2020b al Kosiorek al. [202]] Pumaaola al, Yu et al. 20) Implicit representations have better scaling properties, as usually the output dasolutionano does not directly affect the dimensionality of the learned function. NeRFs (Mildenhall
2021
et al. 2020) generate scenes by learning function that outputs the occupancy and color of points in scene when viewed from a particular direction. By casting rays through a plane and aggregating the output values NeRFs can generate 2D views of an implicitly modeled 3D scenes. NeRFs have obtained superior reconstructions compared to other implict methods, and our model uses NeRFs to represent multiple objects and the background of scene.
Most methods using NeRFs represent scenes monolithically as a single entity. GIRAFFE (Niemeyer & Geiger 2021) is a GAN-based method that represents multiple objects in scene using NeRlis as part of their generator. Their factored representations are amenable to object manipulations. Our model uses rendering pipeline inspired by GIRAFFE. However, representations from existing images, while GIRAFFE does not have an inference mechanism. Ob- jSURF (Stelzner al. 2021) and UORF (Yu et al. 2021b) infer scene representations composed
focus recovering scene
4

---

Under review as a conference paper at ICLR 2022
of multiple objects, each object represented with differently instantiated NeRF. Different from work, they focus on novel view generation. These methods do not explicitly infer the pose of the different objects in the scene. Both methods use ground-truth camera locations and multiple scene views during training, and ObjSURF additionally requires depth annotations.
3.2
CLEECTCENTTICC SCENE MODELS
Segmenting objects in a scene a landmark computer vision task with an extensive literature. Re- cently, there is a line of work on object-centric generative models of scenes (Kosiorek et al. 2021 Burgess et al. 2019 Locatello et al. [2000; Lin et al., 2020 Greff ct al. 2019). These models learn to generate scenes as a composition of multiple objects and a background. When equipped with an inference mechanism, these models learn to segment objects in scene without annotations, driven by their compositional generative process. MONet (Burgess et al. 2019) implements a multi-object VAE that segments object sequentially by infering latents corresponding over parts of the scene not yet attended to iteratively. IODINE (Greff et al. 2019) uses a similar multi-object VAE and per- scene decomposition. Slot-Attention (Locatello
forms multiple rounds of inference to settle ct al. 2020) maps a set of entities, called slots, to image features through multiple rounds of soft attention. The slots compete among themselves to attend to features, making each slot attend to region of the input image. When driven with alpha-compositing decoder, slot attention learns to segment objects in a scene. Our model uses variant of Slot Attention decide on which part of 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object. Object-centric scene models have also been implemented world models, with the goal of simulating dynamics (Lin et al. 2020). By decomposing the scene into objects, these methods can simulate dynamics at an object level, which are usually simpler and shared among objects. Con- trary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.
4 EXPERIMENTS
In this section we showcase the capabilities of INFERNO with three main experiments. First, we demonstrate the interpretability of the scene representations through manipulating scenes and ver- ifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision. Finally, we highlight the usefulness of such representations for downstream tasks by applying our model on the CATER snitch localization task.
4.1 TRAINING SETUP
We use the same training setup for all experiments unless otherwise mentioned. We train mod- els for 400K iterations using the Adam optimizer Kingma & Ba (2014) with a learning rate of up to 16 Vidia V100 GPUs. We use learning
1 x 10-4. We rate warmup (Goyal et al., 2017), which was found to be helpful to avoid optimization issues with Slot Attention. We use weight decay rate À x 10-6 We use three iterations of slot attention during training and evaluation. We set up the number of objects in a scene as the maximum possible number of objects in dataset, i.e. five objects for CLEVR2345, for CLEVR6 and 10 for CATER. Please refer to Appendix B for more details on the experimental setting.
batch size of 128 and
4.2 SCENE INFERENCE
In this section we demonstrate the properties of our scene representation. Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not pos- sible otherwise. These manipulations can be validated by rendering the modified scene represen- tations. Additionally, we verify that decomposing the scene into multiple objects leads to better reconstructions.
First, verify the quality of INFERNO's generations by comparing them two baselines: i) version of our model that does not consider multiple objects, and ii) the GAN method of GI- RAFFE (Niemeyer & Geiger 2021) We perform this comparison the CLEVR-2345 dataset
5

---

Under review as a conference paper at ICLR 2022
Input
Recon.
Addition
Removal
Scale
Forward
Right
Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.
Table 1: Reconstruction error on CLEVR2345 We consider an autoencoder baseline that uses single NeRF object covering the whole scene (NeRF-AE), and compare it to model the test set of CLEVR-2345 NeRF-AE struggles to reconstruct multiple objects accurately. In con- trast, INFERNO obtains much more precise reconstructions under all metrics and allows for object identity/pose manipulations.
Model
MSE (L)
PSNR (t)
SSIM (t)
LPIPS (L)
NeRF-AE
5.14 x 10-4
44.89
59.24 %
168.9
10-3
Ours 1.22 x 10
52.07
72.4
18.93 10 10-3
introduced by GIRAFFE, which contains CLEVR images with 2 to objects. For reconstruction, we compare models using reconstruction metrics including mean-squared error, PSNR and SSIM. We rely on the population metric Frechet Inception Distance (FID) to evaluate the generation quality.
In INFERNO, we generate novel scenes by inferring representations for ground- -ruth images and then manipulating them. To compare to GIRAFFE, we manipulate scenes by adding additional objects and swapping object shapes and appearances across scenes. Manipulations are described in more detail in the Appendix. We highlight that GIRAFFE is unconditional model, while INFERNO generates novel scenes conditioned on existing ones.
We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings. We also conduct this experiment on the CLEVR- 2345 dataset. We validate that our model is able to render ing to training examples, such as scenes having 1 or objects, and verify that the pose manipulations have semantically coherent effects.
scenes not correspond-

---

Under review as a conference paper at ICLR 2022
Table 2: FID on CLEVR2345 We consider our model as NeRF scene generator and compare it to the state-of-the-art. When reconstructing ground-truth images, our model obtains better FID than GIRAFFE. We then consider object manipulations generate novel scenes from existing ones. While adding new objects to a scene slightly increases our FID score, when exchanging object identities across scenes we set a new state-of-the-ar for image generation on CLEVR2345.
Model
FID (L)
GIRAFFE
37.7
Ours Reconstruction
23.5
Ours Add Object
27.2
Ours Swap Object
23.7
Table 1 shows the reconstruction metrics obtained by our model and baseline on the CLEVR2345 dataset. Note that GIRAFFE is a GAN-method that does not have an inference mechanism, and therefore it cannot reconstruct scenes. We observe that the NeRF-AE baseline obtains higher construction error than our regular model, as our object-centric method can make better use of its capacity. In Table 2 compare INFERNO with GIRAFFE using the FID metric. Our model constructions have better FID than the generations of the GIRAFFE. Additionally, our model can perform inference and manipulations scenes by manipulating existing ones. Our model is able to generate novel scenes with additional objects with altered object shapes and appearances, with better FID than GIRAFFE.
existing scenes. We that capability to generate novel
In Figure [ we show some examples of the scene manipulations possible with our model. Given scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales, While some of these manipulations can be performed with regular object-centric models, modifications to the scale and location of the objects are hard to implement without explicitly modeling 3D object pose.
4.3 OBJECT DISCOVERY
Table 3: Object Discovery CLEVR6 INFERNO, despite inferring more complex 3D object seg- mentations without annotations, competitive with the current state-of-the-art 2D object discovery methods CLEVR6.
Model
ARI % (t)
Slot-Attention
98.8 + 0.3
IODINE
98.8 + 0.0
MONet
96.2 + 0.6
Slot MLP
60.4 + 6.6
Ours
96.7 + 0.2
Unsupervised object discovery consists in learning to segment the objects in a scene without using annotations. We test our model on CLEVR6 benchmark, variant of the CLEVR dataset with scenes of up to 6 objects and annotated with 2D object masks. We choose this dataset to compare to previous work in unsupervised object discovery. Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations. We evaluate the quality of the segmentations using the Adjusted Random Index (ART) metric (Rand) 1971). which is a measure of clustering similarity. In line with previous work, compute only the foreground ARI, which does not take into account the background segmentation mask. In particular, we consider object as a different clusters and compare the cluster assignment of each foreground pixel in the original image to its prediction. To determine which pixels correspond to each object in our model we make use of the input segmentation masks predicted by inference mechanism.
Table[]reports the ARI metric our model and different basclines in this task. We observe that IN- FERNO is competitive with state-of-the-ar methods, surpassing MONet and having slightly lower

---

Under review as a conference paper at ICLR 2022
Input
Background
Object
Object
Object
Object
Object
Object
Figure 3: Object Discovery on CLEVR6: INFERÑO identifies the different objects in scene without supervision. For each input image, we show which regions of the input are attended by each object well as the background. We include example of a failed segmentation in the last where one object slot (4) is trying to represent multiple objects at the same time.
ARI than Slot-Attention and IODINE. However, our model learns to segment objects 3D, while the other baselines extract 2D segmentation masks. Figure]]shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed. We also include an ample that our model fails to segment properly multiple objects are segmented by the same object slot, and a single object represented in multiple parts by different slots.
4.4 SNITCH LOCALIZATION
The goal of this experiment is to show that the representation learned by our model useful for the snitch localization task from the CATER dataset (Girdhar & Ramanan 2019). We focus the CATER task involving videos with static camera. The objective is to predict the final position of an object (the snitch) in video. The scenes show multiple objects that move over time, one being the snitch object. The snitch can be occluded and moved around simultaneously by other objects, requiring object tracking and reasoning about dynamics to solve the task.
We follow the experimental setup of Ding et al. 2020). First, we train INFERNO to reconstruct images from the CATER dataset. Once INFERNO trained, we discard the rendering pipeline of our model, and instead feed the scene representations to 12-layer transformer model to predict the final snitch position. Fach object in our representation is given as an input element the transformer. We add learned positional encoding to the object representations based their frame index. Objects in the same frame have the same positional encoding. The transformer last output is given to MLP head that predicts the logits for the 36 possible output positions.
We minimize the sum of the cross-entrope and a L1 loss between the predicted and the true snitch final position. Following (Ding et al. 2020), also consider the of auxiliary SSL loss after pretraining. The SSL loss randomly masks one object fer-frame and tries to predict its representation at the corresponding output. The model minimizes the 1.2 distance between the predicted object
8

---

Under review as a conference paper at ICLR 2022
Table 4: Snitch Localization on CATER. We report Top-1 and Top-5 accuracies for the snitch localization task. Our model outperforms the R3D LSTM and R3D NL LSTM models that learn unstructured representation. It indicates that the structured representation learned by INFERNO is useful for this task. INFERNO pretraining is also critical, showing that the pretraining and not the encoder architecture is a key component. Overall, INFERNO achieves performances close to the state-of-art approaches.
Model
Top-1
Top-5
R3D LSTM
60.2
81.8
R3D + NL LSTM
46.2
69.9
OPNet (extra anns.)
74.8
Hopper
73.2
93.8
Slot-Attention
59.1
88.0
Aloe (w/out SSL loss)
60.1
Aloe
74.0
94.0
Ours (w/out pretraining)
2.91
12.9
Ours (w/out SSL loss)
69.17
87.68
Ours
71.7
88.9
representation and the observed one. The SSL loss is only backpropagated through the transformer and not the inference network.
During training, we randomly samples 40 frames from a video and predict the snitch localization from this one crop. At test time, we randomly sample 10 temporal crops of 40 frames each and average the prediction over the 10 crops as our final prediction. Refer to Appendix for more details about the experiment setup.
Table reports CATER Top-1 and Top-5 accuracies for different methods. We first compare our model that is pretrained to reconstruct image on CATER with randomly initialized encoder. Using a randomly initialized encoder does not perform well. It indicates that INFERNO pretraining, and not the encoder architecture, is key to learn useful representation for the task. We also observe that the additional SSL loss provides regularization and slightly improves the performance our model.
We next compare approach with Aloe (Ding al. 2020). an object-centric baseline, R3D and R3D NL, 3D convolutional models proposed by (Girdhar & Ramanan 2019) and architectures us- ing strong inductive biases toward object tracking such OPNet (Shamsian al., (2000) Hop- per (Zhou et al. 202T). Our model outperforms the R3D and R3D NL models that rely on unstruc- tured representation. This result suggests that the object-centric representation learned by INFERNO useful for the visual reasoning task. Our model also outperforms Aloe, an object-centric method using 2D object representation, when both methods do not use additional SSL loss. However Aloe benefits more from the use of an additional SSL loss. Overall, INFERNO achieves performances close to the state-of-art approaches.
We finally evaluate the performance of slot-attention baseline (Locatelo et al. 2020 in Table The slot-attention baseline first pretrains slot-attention encoder, with similar architecture than our model, by reconstructing CATER frames using mask decoder. It then fine-tunes the encoder using the same procedure than our model to solve the snitch localization task. We observe that our model significantly outperforms the slot-attention model which focus the 2D geometry of the scene. This result supports the advantage 3D-aware representation for solving the CATER task.
CONCLUSIONS
We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model able to discover objects in scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations. Further, the scene representation is useful for visual reasoning downstream tasks such the snitch localization task in CATER.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv:1901. 11300, 2019.
David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Object-based attention for spatio- temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architec- tures. arXiv preprint arXiv:2012.085008 2020.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 3(7),e11, 2018.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and tem- poral reasoning. arXiv preprint arXiv:1910.04744 2019.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Pruceedings of the IEEE/CVF International Conference Computer Vision, pp. 9785-9795, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint 1rXXvv:706.02600 026777 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2433. PMLR, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv: 1910.0384,, 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing of object files: Object-specific integration of information. Cognitive psychology, 24(2):175-2) 2199 1992.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 141.16922..6...... 2014.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo Rezende. Nerf-vae: A geometry aware 3d scene generative model. arXiv preprint arXiv:2104.00580 00587, 2021.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving gen- erative imagination in object-centric world models. In International Conference on Machine Learming, pp. 6140-6149, PMLR, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint 1rXii:2006.15555 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1460-4470 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2022
Thu Nguyen- Phecn, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7588-7597, 2019.
Thu Nguyen-Phuoe Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Block- gan: Learning 3d object-aware scene representations from unlabelled images. arXiv preprint N0w:2002.08988. 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11444 11464, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pullern Recognition, pp. 165-174, 2019.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 10318-10327 2021.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the Ameri- can Statistical association, 66(336):846-855 850, 1971.
Konstantinos Rematas and Vittorio Ferrari. Neural voxel renderer: Learning an accurate and con- trollable rendering tool. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5417-5427, 2020.
Pieter R Roelfsema, Victor AF Lamme, and Henk Spekreijse, Object-based attention in the primary visual cortex of the macaque monkey. Nature, 395(6700):376-388, 1998.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. arXiv preprint arXiv:2007.02442. 2020..
2020.
Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. Learning object permanence from video. In European Conference on Computer Vision, pp. 35-50. Springer, 2020.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awwre neural scene representations. arXiv preprint arXiv: 19060066688 2019.
Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. arXiv preprint arXiv:2006. 09662, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020b.
Elizabeth Spelke, Karen Breinlinger, Kristen Jacobson, and Ann Phillips. Gestalt relations and object perception: A developmental study. Perception, 22(12):1483-150,, 1993.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arx11:21004.01148, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578-4587 2021a.
Hong-Xina Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery object radiance fields. arXiv preprint arXiv:2 (00.::200707900)) 2021b.

---

Under review as a conference paper at ICLR 2022
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu- Martin Renqiang Min, Mub- basir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. arXiv preprint urXiv:2 10574, 2021.
A ADDITIONAL MODEL DETAILS
Our model is composed of five main modules: encoder, slot attention, slot object mapping, NeRF decoder, neural network upscaler. In this section we provide additional details about each of these modules well describe their architecture.
Table 5: Encoder Neural Network
Layer Type
Size
Normalization
Activation
Other details
Conv x 5
64
ReLU
Stride Pad. 1
Conv 5
64
ReLU
Stride Pad. I
Conv X 5
64
ReLU
Stride Pad. I
Conv 5
ReLU
Stride 1 Pad. 1
The goal of the encoder is extract image features. We use an encoder with no down- sampling, as it is typically used with Slot Attention. Details about the encoder architecture are described in Table
Encoder
Table 6: Slot Attention Neural Network
Name
Size
Description
Positional emb.
64
Additive embedding, same size as CNN input features Flattens the spatial dimensions of CNN features
Flatten
QKV MLP
128
Linear layers that map slots and input features to the same dimension
LayerNorm
128
Normalizes the slots/inputs The output of soft-attention goes through a linear layer + GRU
MLP + GRU
128
We employ Slot Attention to map image features to object slots. Slots are sampled randomly from Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots. During training we employ three iterations of slot attention refine the image features slot assignments.
Slot Attention
Table 7: Slot to Object MLP details
MLP Name
Size
Act and Norm.
Description
Obj Pose
7
ReLU, LayerNorm
Slot to translation, scale and rotation
Obj Shape/App.
128
ReLU, NordLayerNorm
Slot to shape and appearance Background slot to its shape/app. fixed pose.
BG Shape/App.
128
ReLU, LayerNorm
Slot to Object Net The background and object slots mapped
parameters using
series of MLP. For each object slot, we first map the object to its pose parameters. We use a 2-layer ML. with LayerNorm map a slot its pose. The size of the hidden dimension the same than
12

---

Under review as a conference paper at ICLR 2022
the output dimension size. We parametrize object pose 7-dimensional tensor. We use three dimensions for the object location along each axis, three dimensions for the scale of the object and single dimension to express rotation of the object along the X axis. These parameters are then mapped to their corresponding 4 x 4 affine transformation matrix for each object. Note that in practice are not modeling rotations in the experimental section.
Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using common 2-layer MLP. The MLPs conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm pa- rameters with single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MIPss, which produce two 128 128ddimensional tensors.
For the background object we only infer shape and appearance, and define its pose to be that of the scene cube. The shape and appearance of the background are inferred through another 2-layer MLP with ReLU and LayerNorm.
Note that our scene representation also admits a camera pose. In our experiments we fix the camera location to look at the scene from standard location (centered and 33 degrees above the Z plane). Other concurrent approaches (Yu et al., 2021b] Stelzner al., (20T)) use ground-truth camera loca- tions to generate novel views, while we focus recovering scene representations without the use of ground-truth annotations.
Table 8: Details about the NeRF MLPs used
MLP Name
Layers
Size
Description
Obj MLP
8
64 ReLU activation, no norm. Skip connection with layer 4. ReLU activation, norm.
BG MLP
16
NeRF MLPs With the object shape and appearance tensors can render them following GRAF (Schwarz et al.) 2020). We use one NeRF for the objects and one NeRF MLP for the back- ground. The details about each NeRF architecture can be found in Table 8 Note that, to render ob- jects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose. To reduce the computational complexity of rendering with many NeRFs, we render scenes at fixed resolution of 16x Instead of rendering RGB pixels, we render feature images with 128 channels. The output of the rendering is then upscaled and mapped to RGB views with neural network upscaler.
Table 9: Neural Upscaler architecture
Layer
Size
Activation
Normalization
Other
Conv 3 x 3
64
ReLU
Instance
Stride=1 Pad=1
Upsample
Nearest Neighbors
Conv 3 x
64
ReLU
Instance
Stride=1 Pad=1
Upsample
Nearest Neighbors
(Only 128px) Conv 3
64
ReLU
Instance
Stride=1 Pad=1
(Only 128px) Upsample
Nearest Neighbors
Conv 3
3
Stride=1 Pad=1
Neural Upscaler The neural upscaler takes the low resolution output of the NeRF MLPs and upscales to the full output resolution. Additionally, it maps the rendered image to RGB space. This module is implemented using a convolutional neural network. We always render the NeRF output at 16px. Consequently, we add additional layers to the neural upscaler depending on the desired output resolution. For most experiments Inference experiments on CLEVR2345 we use a resolution 128px. The architecture the neural upscaler can be found in Table
resolution of 64px, while for the Scene
13

---

Under review as a conference paper at ICLR 2022
B EXPERIMENT DETAILS
B.1 SCENE MANIPULATION
Dataset For generating manipulated scenes consider the CLEVR2345 dataset (Niemeyer & Geiger 2021). Images in the CLEVR2345 dataset contain from to objects. We use the original train and test splits. Images resized 128x 128x128 pixels and RGB values normalized in the (0, 1] range.
Training We use batch size of 128 and train our model for 400k iterations. We use Adam with learning rate of x 10 and weight decay x 10 We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects. We the neural upscaler with additional layers to upscale to 128px. Additionally, for this experiment we use an additional LPIPS loss. We use the LPIPS metric computed by an AlexNet network, and we add this loss to our regular MSE loss. We weight the LPIPS loss by a factor of 100, so that it has comparable order of magnitude to MSE loss.
Manipulations
Manipulations are done as follows:
Substraction: We randomly delete up to two of the object slots.
Addition: We randomly add an object slot from another scene.
Scale: We reduce the scale (in all XYZ axis) of one of the objects in the substraction scene.
Forward: We manipulate the pose vector of one object in the substraction scene and move it forward on the Z axis.
Right: We manipulate the pose vector of one object in the substraction scene and move it forward on the X axis.
Additionally, we consider the Swap transformation for Table [ This transformations modifies a scene by replacing the object shape and appearance vectors with those of an object from another scene.
Mean-Squared Error, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) and the LPIPS metrics.
Metrics To compare reconstructions
MSE measure the average squared difference between pixel values.
MSE x')
xx x²)²
(5)
PSNR is a metric commonly used in signal processing.
PSNR( xx) = -10 1o (MSE(r,x")
(6)
SSIM (Wang et al. (004)) provides scores more aligned with human perception, specially under the presence of image noise. Scores are computed convolutionally by applying a kernel over images, which are then contrasted.
LPIIS (Zhang et al. 2018) computes differences neural network activations for two images. perceptual metric that has been shown to have higher correlation to human perception than other metrics not based on neural networks.
To compare populations of generated images
the Frechet Inception Distance (Heusel et al. 2017). The Frechet Inception Distance embeds images into a neural network space and then fits a Gaussian distribution to the generated and ground-truth activation statistics. The score is obtained by then computing the Frechet distance between the two. Note that other metrics such as Inception Score applicable for the CLEVR2345 since there are well-defined classes.
14

---

Under review as a conference paper at ICLR 2022
B.2 OBJECT DISCOVERY
Dataset For object discovery we consider the CLEVR6 dataset. We use the original CLEVR6 dataset and extract the images from TFRecord files available at this URL, We use the original train- ing/test split, using the first 70% images for training and the remaining ones for test. We take crop between pixels [29, 221] and [64, 256], for the height and width respectively, and then resize the crop to 64px. We normalize the value of the images between [0, 1]. To generate the CLEVR6 dataset, we keep only those images that have at maximum objects according to the annotation files.
Training We use a batch size of 128 and train our model for 400k iterations. We use Adam with learning rate of 1 x 10-4 and weight decay 1 10-6, We use 6 objects and rely on the model to not use additional slots when needed.
Metrics We follow previous work (Greff et al. 2019) and use the Adjusted Rand Index (ARI) (Rand 197T) evaluate cluster assignments in object discovery. ARI scores range from (random assigment) to I (perfect match). As in previous works, we do not consider segmentation mask for the background.
B.3 VISUAL REASONING ON CATER
For the visual reasoning task, we consider the CATER dataset and uses 5K videos that do not have camera motion. All the videos are resized to a 64x64 resolution.
B.3.1 PRETRAINING
We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train on the CATER dataset for additional 100k iterations to speed-up training, as the iteration time of model with 10 objects is larger. We use a batch size of 128 and we use Adam with a learning rate of 1 10-4 and weight decay 1 10-6, When training CLEVR6 we use 6 object slots, while when training on CATER we use 10 object slots.
B.3.2 FINETUNING
After pretraining, we finetune the INFERNO encoder to the supervised task of snitch localization. We discard the rendering pipeline of our model, and instead feed the inferred object slots represen- tations to transformer that aims at predicting the final position of the snitch.
To predict the snitch, we consider a 12 layers transformer with the hidden dimension of 128 which takes the slot representation as input. The transformer treats each object as input element. A learned positional embedding is added each slots representation based on their frames index, i.e. the position of the objects is the same within frame. The final output to the transformer given
I layer MLP head with an hidden dimension of 128. It outputs 36 logits that correspond to possible snitch location. We minimize the sum of the cross-entropy between the predicted position and the true target and the I1 loss between the prediction and target.
We optionally use a SSL loss similar to Ding al. (2020). The SSL loss randomly masks one object per-frame and tries to predict its representation at the corresponding output. The model minimizes the L2 distance between the predicted object representation and the observed one. The SSL loss is only backpropagated through the transformer and not the encoder. We weight the SSL loss by factor of 1.0e 3.
We use Adam optimizer to minimize the loss. The initial learning rate is set to the 1.0e 4 and gradually decreased to 1.0e 6 using a cosine learning rate decay. Similarly, we use a initial weight decay of 1.0c - that increases to 1.0e 500 epochs. We don't 't make use of learning rate decay.
using cosine schedule. Our model is finetuned for
During training, randomly samples 40 frames from video and predict the snitch localization from this one crop. At test time, we randomly sample 10 temporal crop of 40 frames each and average the prediction over the 10 crops as final prediction.
15

---

Under review as a conference paper at ICLR 2022
C
ADDITIONAL VISUALIZATIONS
Input
Recon.
Input
Recon.
Input
Recon.
Figure 4: Additional reconstructions on CLEVR2345.
We have included additional manipulations one scene in GIF format in the supplementary mate- rial. We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in circle.
16

---

Under review as a conference paper at ICLR 2022
Input
Add
Remove
Input
Add
Remove
Figure 5: Additional object additions and removals CLEVR2345. For each scene, show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with number of objects not seen during training (2-5 objects).
17

---

Under review as a conference paper at ICLR 2022
D NOVEL VIEW SYNTHESIS
In this section we synthesize novel views of a scene by modifying the camera posc. For cach example we show the input image, reconstruction, then two images a result of moving the camera 15 in the azimuth axis, and two images as result of zooming the scene. While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views scene
Input
Recon.
Rot. +15 deg. Rot. -15 deg.
Zoom x1.5
Zoom x2
Figure 6: Novel view synthesis CLEVR2345. For each scene, move the camera = on the azimuth axis. Additionally, we zoom in the scene twice. While our model trained with fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of scene.
18

---

Under review as a conference paper at ICLR 2022
E
NERF OUTPUT
In this section we show the raw outputs of the NeRF function. These outputs show scene views rendered at low resolution, which are then upscaled with neural network. While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene. NeRFs rendered low resolution to ease the computational costs, the time and memory requirements of rendering with a NeRF are linearly correlated with the number of casted rays/pixels in the output image.
Input
Recon,
NeRF out
Input
Recon.
NeRF out
Figure 7: NeRF outputs CLEVR2345. For each input scene show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.
19

---

Under review as a conference paper at ICLR 2022
F SHAPE APPEARANCE DISENTANGLEMENT
In this section we show examples of the disentanglement of shape and appearance of objects. To demonstrate this property our representations, we first reconstruct a given scene, and we then randomly change the appearance vector of one of the objects in the scene by the appearance vectors of other objects in the same scene. While the shading and color the object changes, the overall shape remains the same.
Input
Recon.
App.
App.
Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.
20

---

Under review as a conference paper at ICLR 2022
G OCCUPANCY MAPS
In this section we show the occupancy maps for each rendered object. The occupancy maps are obtained by integrating rays going from the image plane the scene according the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding a single object instance. NeRF outputs are rendered at low resolution and then upscaled to full resolution with neural network. To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.
Input
Recon.
Obj.
Ohj.
Obj.
Obj 4
Obj. 5
Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the recon- structed image well each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the single object
scene. We observe that each object rendering part of the scene corresponding instance. Outputs of the NeRF function have low resolution and are upscaled by neural network. The last row shows example of bad grouping of two objects in the scene.
21


Reviews:
[{"review_id": "wRi1dlaKYQY", "reviewer": "Reviewer_k83T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles a very challenging problem, is well-written and easy to follow, and has a good set of experiments for testing object-centric representations. However, my main concern is that the experiments did not focus on testing the '3D-ness' of the inferred representations.\n- For scene segmentation, reconstruction, and manipulation, each scene is tested only on a single camera viewpoint. It is unclear whether the learned representations can capture the object appearance from viewpoints different from the input view. Previous work [1, 2] demonstrates this through novel view synthesis.\n- The results on the CATER snitch localization task did not strongly support that the '3D' representations learned by the proposed model are more beneficial than 2D representations. One reason might be that this task relies more on long-term reasoning than on 3D reasoning. The authors may want to consider some other tasks that can clearly show the benefit of having 3D representations, like the ones used in [2]. Also, instead of comparing to Aloe, a more direct comparison would be to compare with Slot Attention combined with the same 12-layer transformer used for the proposed model.\n\n**MINOR COMMENTS**\n- The rendering function is denoted $g_\\gamma$ in Equation 1 but $g_\\tau$ in Section 2.1.\n- From the paragraph directly below Figure 1, it seems the model can infer the camera location $c$. However, this was not demonstrated in any of the experiments.\n- The object NeRFs are shared, which may only be able to deal with similar objects. The paper could be strengthened by experiments on more complex scenes, like the ones used in [1].\n- What is the reference frame for the pose vectors? It is mentioned in Appendix B.1 that the manipulations are done by moving the objects forward on the Z or X axes. How are the XYZ axes defined? I would imagine that under different camera viewpoints, the same manipulation would have different effects, but the results in Figure 2 seem quite consistent.\n\n[1] [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://arxiv.org/abs/2104.01148).\n\n[2] [ROOTS: Object-Centric Representation and Rendering of 3D Scenes](https://arxiv.org/abs/2006.06130).", "summary_of_the_paper": "The paper aims to decompose a scene into objects and infer the representations of 3D occupancy, color, and pose for each object from a single image of the scene without supervision. To this end, the paper proposes an autoencoding solution by combining the Slot Attention encoder with the GIRAFFE decoder. Each object is represented as a Neural Radiance Field (NeRF) additionally parameterized by the latent variables inferred from the encoder. The decoder then compositionally renders the objects. The experiments show that the proposed model (1) achieves competitive 2D segmentation performance on CLEVR6, (2) supports object-wise scene manipulation, and (3) outperforms non-object-centric methods on CATER snitch localization when combined with a powerful transformer.", "summary_of_the_review": "I recommend reject, because the main claim, learning 3D representations, is not well supported by the experiments, and the benefit of the learned representations over prior work is not well demonstrated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qe7h164Xe4", "reviewer": "Reviewer_noTD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ The paper is clearly written and easy to follow.\n+ It is novel and interesting to see object-centric learning models to infer 3D poses.\n+ Validating the usefulness in downstream visual reasoning task is a plus.\n\nWeakness:\n\nMy main conern is in the insufficient experiments that do not support the main contributions, i.e., the experiments do not demonstrate the benefits of the proposed 3D, structured object representations.\n\nAs for representations being structured (i.e., disentangled shape, appearance and poses):\n- For disentanglement of shape and appearance, there is no experiments demonstrating this point. For example, this can be easily shown by swapping appearances of two objects in the scene while keeping their shapes unchanged, and vice versa.\n- For disentanglement of 3D pose, the experiments include scaling and translation. However, there is no comparison to 2D baseline methods, which are able to do scaling and translation (at least the experiments on moving to the \"right\").\n\nAs for representations being 3D, there is no comparisons to show the benefits of 3D representation over 2D. In segmentation experiments, the results are worse than 2D baselines. In the visual reasoning task, the performance is worse than Aloe. Perhaps Aloe is better because it is designed for that task, but there is neither comparisons to 2D general object-centric baselines (such as original slot attention) or unstructured baselines.\n\nOther minor weaknesses include:\n- No ablation study on the model components to show the benefits of proposed technical contributions, such as the effect of regressing pose vs. not regressing pose (I suppose this should affect background decomposition because that is the only inductive bias that separates background slot from foreground slots).\n- It would be good to demonstrate on datasets other than CLEVR.", "summary_of_the_paper": "This paper proposes a novel unsupervised scene decomposition model that infers object shapes, appearances and 3D poses. The benefits over existing models are the structured, 3D object representations which allows to manipulate objects in the scenes such as moving and replacing objects. This paper also shows that the inferred object representations can be used in a visual reasoning task.", "summary_of_the_review": "Although this paper is interesting in the ability to infer 3D properties of objects, the fundamental flaws in experiments are the main reasons that I recommend revision and resubmission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "B7uvpZ9G0WR", "reviewer": "Reviewer_3rCh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. It is interesting to use NeRFs to represent objects in a scene, which encourages the understanding of 2D visual scene taking account into its 3D structure.\n2. Explicitly modeling the 3D structure of 2D scene may help better deal with occlusion, ambiguity from similar appearance which can be a big challenge in 2D scene understanding.\n3. It looks great to show how learned object features can be used for downstream tasks.\n\nWeaknesses:\n1. The technical novelty is somewhat limited. The paper mainly combines the Slot-attention and NeRF. Personally, it is hard to identify the exciting point(s). \n\n2. The motivation of this work seems not well justified. It is claimed that the model is proposed to infer 3D representation for objects that are interpretable and amenable. But the experiments presented with such 3D representation cannot show its superiority and it is\nhard to see the necessity of 3D object representation in this case considering the task presented can also be fulfilled with 2D object representation. In terms of the interpretability and manipulability of latent code, this paper presents objects removal/addition, object scaling and object translation. All of the above operations can actually be done just by explicitly modeling 2D object pose such as AIR (NIPS'16 paper). In terms of scene understanding in the form of object segmentation, this model cannot beat Slot Attention. In terms of downstream tasks, this model is not compared with other object-centric models. Thus, based on your current experiments, the motivation for learning such 3D representation for objects is not well justified.\n\n3. Is it appropriate to use NeRF? If I understand correctly, NeRF learns the radiance field of a scene with 2D images rendered from different viewpoints. This model tries to recover the object radiance field from a single view of the objects. But the model training\nobjective can actually be fulfilled by just learning a single-view of the scene since there is only one image for each scene to generate reconstruction loss.\n\n4. From my understanding, the 3D rendering model in this paper seems more similar to a decoder in VAE-wise architecture. It could be more convincing to provide some novel view synthesis (not novel scene) to claim it as a NeRF. For example, you may consider\nobject rotation.\n\n5. All experiment datasets are a bit too simple. It is unclear how good/bad the proposed model will perform on realistic 3D scenes such as LLFF dataset or ScanNet. Otherwise, it is hard to see the real impact of this proposed method.\n", "summary_of_the_paper": "This paper proposes a model to infer structured 3D object representations from a 2D scene in an unsupervised fashion so as to represent the visual scene in an object-centric way. Specifically, the inference part adopts a similar mechanism as Slot Attention to derive object\nslot latent code, and then maps slot latent code to 3D object representations with MLP. The rendering part takes the idea from 3D neural rendering where a shared NeRF function is used to represent all objects excluding the background. Rendering is performed by querying NeRF with 3D location, 2D view direction as well as object latent to get object color value and occupancy value. Object rendering is composed into scene rendering according to location derived at the inference stage and weighted by occupancy value.\n\nTo sum up, the paper interprets a 2D visual scene with 3D object-centric representation. With the existing 2D object-centric scene segmentation method and 3D neural rendering approach, it achieves comparable segmentation performance and derives manipulable object representation.", "summary_of_the_review": "Look forward to the response from authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "59SiWHb1LI", "reviewer": "Reviewer_G78k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* an interesting combination of slot attention and NeRF mixture models.\n* paper is nicely presented\n* results are nice, though see below\n\nWeaknesses:\n\nWhile I like the paper in general I feel there are several points missing which are probably worth addressing to:\n\n* The objects are transformed using the pose - but that means the appearance is transformed as well - this could be a problem if some of the appearance is pose dependent, for example shading will change, or reflections. Can the authors comment how this is handled here? my guess is that the post rendering network (the upscaling one) takes some of this burden but that's probably not ideal in terms of representation.\n\n* How does the model perform when scenes are rendered from other viewpoints? if the resulting representation is indeed 3D it should be no problem to render the scene from other viewpoints - is that the case here? if not, why? (again, my guess is the upscaling network may be problematic here as well)\n\n* I see that the masks used for estimating segmentation performance (Figure 3) are from the slot attention mechanism - what happens if you use the masks as decoded from the slots (i.e. render the densitiy of each slot separately). This would be more convincing to demonstrate these are \"3D\" segmentation.\n\n* I feel the experimental validation is quite limited - using only a single dataset (even if it's two variants) is not very convincing as to the generality of the method. How does it work on other datasets? more/less complex.\n\n* I would be happy to see more discussion of the post rendering / upscaling network - this seems like a cruicial part and is not discussed enough.", "summary_of_the_paper": "This paper proposes a model which is able to segment 3D scenes into objects by a combination of slot-attention (For inference) and a mixture of object NeRF functions which mix together (in 3D) to compose a scene. The method receives a single input image (with the camera coordinates though these are fixed) and extracts a set of slots - one slot for each object. These slots are decoded using a NeRF renderer: one part (the shape) generates the density, one (the appearance) generates the colours and one (the pose) transforms the points of the object to the appropriate pose in scene space. Results are demonstrated on CLEVR data as well as CATER (which is visually very similar) and some downstream tasks.", "summary_of_the_review": "All in all a nice paper but I think it suffers from some drawback as discussed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for learning a decomposition of a scene into 3D objects using single images without pose annotations as training data. The model is based on Slot Attention and NeRF. Results are demonstrated on CLEVR and its variants. 

The reviewers point out that the method is reasonable and the paper is quite good, but even after considering the authors' feedback agree that the paper is not ready for acceptance. In particular, the key concern is around experimental evaluation - that it is performed on one dataset (and variants thereof) and that the evaluation of the 3D properties of the model is not sufficiently convincing: it does not outperform 2D object learning methods on segmentation and is not compared to those on "snitch localization".

Overall, this is a reasonable paper, and the results are promising but somewhat inconclusive, so I recommend rejection at this point, but encourage the authors to improve the paper and resubmit to a different venue.

(One remark. The paper makes a point of not using any annotation. It is technically true, but in practice on CLEVR unsupervised segmentation works so well that it's basically as if segmentation masks were provided. If the authors could demonstrate that their method - possibly with provided coarse segmentation masks - works on more complex datasets, it would be a nice additional experiment)
Decision: Reject