Object Scene Representation Transformer
Authors: Mehdi S. M. Sajjadi
Daniel Duckworth
Aravindh Mahendran
Sjoerd van Steenkiste
Filip Pavetic
Mario Lucic
Leonidas Guibas
Klaus Greff
Thomas Kipf
Venue: NeurIPS 2022 Accept (2022)
Keywords: novel view synthesis
scene decomposition
transformer
slot attention
unsupervised decomposition
representation learning
neural rendering
scene representations

Abstract: A compositional understanding of the world in terms of objects and their geometry in 3D space is considered a cornerstone of human cognition. Facilitating the learning of such a representation in neural networks holds promise for substantially improving labeled data efficiency. As a key step in this direction, we make progress on the problem of learning 3D-consistent decompositions of complex scenes into individual objects in an unsupervised fashion. We introduce Object Scene Representation Transformer (OSRT), a 3D-centric model in which individual object representations naturally emerge through novel view synthesis. OSRT scales to significantly more complex scenes with larger diversity of objects and backgrounds than existing methods. At the same time, it is multiple orders of magnitude faster at compositional rendering thanks to its light field parametrization and the novel Slot Mixer decoder. We believe this work will not only accelerate future architecture exploration and scaling efforts, but it will also serve as a useful tool for both object-centric as well as neural scene representation learning communities.

Object Scene Representation Transformer
Mehdi S. M. Sajjadi, Daniel Duckworth", Aravindh Mahendran', ,Sooerd van Steenkiste', Filip Pavetić, Mario Lučié, Leonidas J. Guibas, Klaus Greff, Thomas Kipf" Google Research
Abstract
A compositional understanding of the world in terms of objects and their geome- try in 3D space is considered a cornerstone of human cognition. Facilitating the learning of such representation in neural networks holds promise for substantially improving labeled data efficiency. As key step in this direction, we make progress the problem of learning 3D-consistent decompositions of complex scenes into individual objects in an unsupervised fashion. We introduce Object Scene Rep- resentation Transformer (OSRT), a 3D-centric model in which individual object representations naturally emerge through novel view synthesis. OSRT scales to significantly more complex scenes with larger diversity of objects and backgrounds than existing methods. At the same time, it is multiple orders of magnitude faster at compositional rendering thanks its light field parametrization and the novel Slot Mixer decoder. We believe this work will not only accelerate future architecture exploration and scaling efforts, but it will also serve as useful tool for both object-centric as well as neural scene representation learning communities.
Introduction
As humans, interact with a physical world that is composed of macroscopic objects' situated in 3D environments. The development of an object-centric, geometric understanding of the world is considered a cornerstone of human cognition [32]: we perceive scenes terms of discrete objects and their parts, and our understanding of 3D scene geometry is essential for reasoning about relations between objects and for skillfully interacting with them.
Replicating these capabilities in machine learning models has been a major focus in computer vision and related fields [12. 22, 34], yet the classical paradigm of supervised learning poses several challenges: explicit supervision requires carefully annotated data at a large scale, and is subject to obstacles such as rare novel object categories. Further, obtaining accurate ground-truth 3D scene and object geometry is challenging and expensive. Learning about compositional geometry merely by observing scenes and occasionally interacting with them- the simplest case by moving camera through a scene withhut relying on direct supervision is an attractive alternative.
without
As objects in the physical world situated in 3D space, there has been growing interest in combining recent advances in 3D neural rendering [25] and representation learning [7, 20] with object-centric inductive biases jointly learn represent objects and their 3D geometry without direct supervision [26, 33, 40). A particularly promising setting for learning both about objects and 3D scene geometry from RGB supervision alone is that of novel view synthesis (NVS). where the task is to predict a scene's 's appearance from unobserved view points. This task not only encourages model learn geometricallyyconiistee
representation of scene, but has the potential to serve as
an additional inductive bias for discovering objects without supervision.
Website: put-paper.githubble github.o.. Correspondenee: oart@msaj jaiaijjad addi. com. 'Equal technical contribution. We use the term "object" in very broad sense, capturing other physical entities such as embodied agents.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Query Rays
SRT Encodee
Siot Atterntion
Slot Mixer
Novel Scene Input Views
Set-Latent Scene Representation
Slot Representation
Ceject-Decompoee Novel
Figure I: OSRT overview A set input views of a novel scene are processed by the SRT Encoder, yielding the Set-Latent Scene Representation (SLSR). The Slot Attention module converts the SLSR into the object-centric Slot Scene Representation. Finally, arbitrary views with DD-oosiittent object instance decompositions
efficiently rendered by the novel Slot Mixer. The entire model is trained end-to-end with an 1,2 loss and no additional regularizers. Details in Sec.
Prior methods combining object-centric inductive biases with 3D rendering techniques for NVS [26, 33, 40] however fail to generalize to scenes of high visual complexity and face significant shortcoming in terms computational cost and memory requirements: common object-centric models decode each object independently. adding significant multiplicative factor to the already expensive volumetric rendering procedure which requires hundreds of decoding steps. This require- ment of executing thousands of decoding passes for each rendered pixel is major limitation that prohibits scaling this class methods to more powerful models and thus to more complex scenes.
In this work, we propose Object Scene Representation Transformer (OSRT), an end-to-end model for object-centric 3D scene representation learning. The integrated Slot Attention [23] module allows to learn scene representation that is decomposed into slots, each representing object part of the background. OSRT is based on SRT, utilizing light field parameterization of space to predict pixel values directly from latent scene representation in single forward pass. Instead of rendering each slot independently, we introduce the novel Slor Mixer, a highly-efficient object- aware decoder that requires just single forward pass per rendered pixel, irrespective of the number of slots in the model. In summary, OSRT allows for highly scalable learning object-centric 3D scene representations without supervision.
Our core contributions are as follows:
We present OSRT, model for 3D-centric representation learning, enabling efficient and scalable object discovery in 3D scenes from RGB supervision. The model is trained purely with the simple L.2 loss and does not necessitate further regularizers additional knowledge such as depth maps [33] or explicit background handling [40].
As part OSRT, we propose the novel Slot Mixer, a highly efficient object-aware decoder that scales to large numbers of objects in scene with little computational overhead.
We study several properties of the proposed method, including robustness studies and advantages using NVS to facilitate scene decomposition in complex datasets.
a range of experiments from easier previously proposed, to complex many-object scenes, we demonstrate that OSRT achieves tate-offthee-rt object decomposition, outperforming prior methods both quantitatively and terms of efficiency.
2 Method
We begin this section with a description of the proposed Object Scene Representation Transformer (OSRT) shown in Fig. 1 and its novel Slot Mixer decoder, and conclude with a discussion of possible alternative design choices.
2.1 Novel view synthesis
The starting point for our investigations is the Scene Representation Transformer (SRT) [29] geometry-free novel view synthesis (NVS) backbone that provides instant novel-scene generalization and scalability to complex datasets. SRT is based encoder-decoder architecture.

---

A data point consists of a set of RGB input images {I, E [x from the same scene." A convolutional network CNN independently encodes each image into a feature map, all of which are finally flattened and combined into single set of tokens. An Encoder Transformer & then performs self-attention on this feature set, ultimately yielding the Set-Latent Scene Representation (SLSR)
{2, RR Eo( (O((NN((LL)))
Novel views are rendered using 6D light-field parametrization r - (o, d) of the scene. Each pixel to be rendered is described by the camera position and the normalized ray direction d pointing from the camera through the center of the pixel in the image plane. As shown in Fig. (left), the Decoder Transformer D uses these rays queries to attend into the SLSR, thereby aggregating localized information from the scene, and ultimately produces the RGB color prediction
C(r) Do(r {2₂}).
Given dataset of images (Ie. from different scenes indexed by the model is trained end-to-end using an L2 reconstruction loss for novel views:
arg min
Σ E (lc(_lC(rr))))))))r( - 1sth
- (r)
(3)
2.2 Scene decomposition
SRT's latent representation, the SLSR, has been shown to contain enough information to perform downstream tasks such as semi-supervised semantic segmentation [29]. However, the size of the SLSR is directly determined by the number and resolution of the input images, and there is no clear one-to-one correspondence between the SLSR tokens and objects in the scene.
To obtain an object-centric scene representation, we incorporate the Slot Attention [23] module into architecture. Slot Attention converts the SLSR {z,} } into the Slot Scene Representation (SlotSR), set of object slots {s,, ] Different from the size of the SLSR, the size N of the SlotSR is chosen by the user.
We initialize the set of object slots using learned embeddings (s,. E ER²). The Slot Attention module then takes the following form for learned linear projections W.. Wg, and W, and learned update function Ug:
Snz 2j=1
Ang Woz,
Wwz)) Σ e W.n;)
Sy = Uo
with An.j
(4)
> An.j
The attention matrix A is used to aggregate input tokens using a weighted mean. Different from commonly used cross-attention [36], it normalized over the output axis, i.e., the set of slots, instead of the input tokens. This enforces an exclusive grouping of input tokens into object slots, which serves as an inductive bias for decomposing the SLSR into individual per-object representations. Following Locatello et al. [23], we use gated update for 2g in the form of GRU [6] followed by residual MLP and we apply LayerNorm [2] both inputs and slots.
2.3 Efficient object-centric decoding
To be able to extract arbitrary-view object decompositions from the model, we propose the novel Slot Mixer (SM): powerful, yet efficient object-centric decoder. The SM module is shown in Fig. (center) and consists of three components: Allocation Transformer, Mixing Block, and Render MLP.
Allocation Transformer. The goal of the Allocation Transformer is to derive which slots are relevant for the given ray r (o, d). In essence, this module derives object location and boundaries and resolves occlusions. Its architecture is similar to SRT's Decoder Transformer. It is a transformer that uses the target ray r as the query repeatedly attend into and aggregate features from the SlotSR:
x = Do {s
(5)
Most compute in the Allocation Transformer is spent on the query, allowing it to scale gracefully to large numbers of objects in the SlotSR. However, unlike in SRT, its output is not used directly for the RGB color estimate.
"The images may optionally contain pose information, which omit here for notational clarity.
3

---

SRT Decoder
Slot Mixer Decoder
Spatial Broadcast Decoder
Allocation Trensformer
Slot
Slat N
Mixing Block
Decoder
Weighted Mean
W
RGB,
RGB,
RGB
Mix
Alpha Blending
RGB
RGB
Figure Decoder architectures Comparison between SRT, the novel Slot Mixer (SM), and Spatial Broadcast (SB) decoders. The SRT decoder uses an efficient Transformer that scales gracefully to large numbers of objects, but fails to produce novel-view object decompositions. The commonly used SB model decodes each slot independently, leading to high memory and computational requirements. The proposed SM decoder combines SRT's efficiency with SB's object decomposition capabilities. Details in Secs. 2.3 and 2.4.
Mixing Block. Instead, the resulting feature x is passed to the Mixing Block, which computes normalized dot-product similarity w with the SlotSR matrix S E RNx iee, the slots in arbitrary. but fixed order. This similarity is then used to compute weighted mean of the original slots:
Q -Qx, K S
softmax(K" Q),
s= s=w'S.
where Wo and WK are learned linear projections. Notably, the weight w, of each slot is scalar, and unlike standard attention layers, no linear maps are computed for the slots, Le. the Allocation Transformer is solely responsible for mixing the slots, not for decoding them. The slot weights can be seen as novel-view object assignments that are useful for visual inspection the learned representation and for quantitative evaluation, see Sec.
Render MIP. Finally, the Render MLP decodes the original query ray r conditioned on the weighted mean of the SlotSR into the RGB color prediction:
Crr) Mo(r 5)
We call the resulting model shown in Fig. with the SRT encoder, incorporated Slot Attention module and the Slot Mixer decoder Object Scene Representation Transformer (OSRT). All parameters trained end-to-end using the L.2 reconstruction loss as in Eq. (3).
2.4 Alternative decoder architecture
Our novel Slot Mixer differs significantly from the standard choice in the literature where Spatial Broadcast (SB) decoders [37] (Fig. 2, right) are most commonly used in conjunction with Slot Attention [23, 33, 40]. We present an adaptation thereof to OSRT as an alternative to the SM decoder.
In SB decoders, the query ray is decoded for each slot s,, independently using shared MLP M:
Co,r: Cn,r MM sn))
For each ray r, this produces color estimates C, E R and logits a, R with each value corresponding to a different slot. The final RGB color estimate is then calculated by normalized, weighted mean:
w = softmax(ax),
C(r) = wTc. TT
In the SB decoder, the slots therefore compete for each pixel through softmax operation.
We note major disadvantage of this popular decoder design: it does not scale gracefully to larger numbers of objects or slots, as the full decoder has to run on each slot. This implies a linear increase in memory and computational requirements of the entire decoder, which is often inhibitive.

---

especially in training. In practice, most pixels are fully explained by a single slot, i.e. almost all of the compute of the decoder is spent on resolving the most prominent slot, and the rest go unused.
The proposed Slot Mixer solves this by employing the scalable Allocation Transformer for blending between slots more efficiently, while only single weighted mean of all slots must be fully decoded by the Render MLP, We further investigate the choice of decoders empirically in Sec. 4.2.
3 Related works
Neural rendering. Neural rendering is a large, promising field that investigates the use of machine learning for graphics applications [34]. Recently, NeRF [25] has sparked a renewed wave of interest in this field by optimizing MI. to parameterize a single volumetric scene representation and demonstrating rhoto-realistic results on real-world scenes, later also for uncurated in in-the-wild data [24]. Further methods based on NeRF are able to generalize across scenes by means of reprojecting 3D points into 2D feature maps [35, 39], though they lack global 3D-based scene representation that could be readily used for downstream applications.
Meanwhile, there have been early successes with global latent models [15, 31J, though these rarely scale beyond simple datasets of single oriented objects uniform backgrounds [20]. Altemattve approaches produce higher-quality results by employing a computationally expensive auto-regressive generative mechanisms that does not produce spatially or temporally consistent results [28]. Recently, the Scene Representation Transformer (SRT) [29] has been proposed as global-latent model that efficiently scales highly complex scenes by means of replacing the volumetric parametrization with light field formulation.
Object-centric learning. Prior works on object-centric learning, such as MONet (3), IODINE [9), SPACE [21] and Slot Attention [23] have demonstrated that it is possible learn models that decompose images into objects using simple unsupervised image reconstruction objectives, These methods typically use a structured latent space and employ dedicated inductive biases in their architectures and image decoders. To handle depth and occlusion in 3D scenes, these methods typically employ a generative model which handles occlusion via apha-bbennding [9, 23] or by, e.g.. generating objects ordered by their distance to the camera [1, 21]. We refer to the survey by Greff al. [10] for an overview. Recent progress in this line of research applies these core inductive biases to work images of more complex scenes [30] or video data [17, 19]. In our OSRT model, we make use of the ubiquitous Slot Attention [23] module because of its efficiency and effectiveness.
3D object-centric methods. Several recent works have extended self-supervised object-centric methods to 3D scenes [5, 33, 40]. One of the first such approaches is ROOTS [5]. probabilistic generative model that represents the scene in terms of 3D feature map, where each 3D "patch" describes the presence (or absence) of an object. Each discovered object is independently rendered using a GQN [7] decoder and the final image is recomposed using Spatial Transformer Networks [14].
Further prior works that are the most relevant to our method are ObSuRF [33] and uORF [40], both combining learned volumetric representations [25] with Slot Attention [23] and Spatial Broadcast
decoders [37]. Both methods have been shown to be capable of modeling 3D scenes of slightly higher complexity than CLEVR [16].
ObSuRF's architecture is based on NeRF-VAE [20]. It bypasses the extraordinary memory require- ments of the Spatial Broadcast decoder combined with volumetric rendering during training by using ground-truth depth information, thereby needing only 2 samples per ray instead of hundreds. During inference, at the absence of ground -truh depth information, ObSuRF however suffers from the expected high computational cost of object-centric volumetric rendering.
uORF is based on encoder-decoder architecture that explicitly handles foreground (FG) and background (BG) separately through a set of built-in inductive biases and assumptions on the dataset. For instance, the dedicated BG slot is parameterized differently, and FG slots are encouraged to only produce density inside a manually specified area of the scene. The model is trained using a combination of perceptual losses, and optionally also with adversarial loss [8].
In order to obtain the results for ObSuRF and uORF, we used the available official implementations, and consulted the respective authors of both methods to ensure correct adaptation to new datasets.
5

---

Finally, a separate line of work considers purely generative object-centric 3D rendering without the ability to render novel views of a specifically provided input scene. GIRAFFE [26] addresses this problem by combining volumetric rendering with GAN-based [8] loss. It separately parameterizes object appearance and 3D pose for controllable image synthesis. Inspired thereof, INFERNO [4] combines a generative model with Slot Attention-based inference model to learn object-centric 31) scene representations. We do not explicitly compare to INFERNO as it is similar to ObSuRF and uORF, while only shown capable of modeling CLEVR-like scenes of lower visual complexity.
4 Experiments
To investigate OSRT's capabilities, we evaluate it on a range of datasets. After confirming that the proposed method outperforms existing methods on their comparably simple datasets, we move on to a more realistic, highly challenging dataset for all further investigations. We evaluate models by their novel view reconstruction quality and unsupervised scene decomposition capabilities qualitatively and quantitatively. We further investigate OSRT's computational requirements compared to the baselines and close this section with some further analysis into which ingredients are crucial to enable OSRT's ss unsupervised scene decomposition qualities in challenging settings.
Setting and evaluation metries. The models are trained in novel view synthesis (NVS) setup: on dataset of scenes, we train the models to produce novel views of the same scene parameterized by target camera poses. For evaluation, we run the models on a held-out set of test scenes and render multiple novel views per scene.
As quantitative metrics, we report PSNR for pixel-accurate reconstruction quality and adopt the standard foreground Adjusted Rand Index (FG-ARI) [13, 27] to measure object decomposition. Crucially, we compute FG-ART on all rendered views together, such that object instances must be consistent between different views. This makes our metric sensitive to 3D-inconsistencies that may especially plague light field models which do not explicitly enforce this in contrast to volumetric methods. We analyze and discuss this choice further in Sec. 4.2.
For qualitative inspection of the inferred scene decompositions, we visualize for each rendered pixel the slot with the highest weight w, and color-code the slots accordingly.
Datasets. We run experiments on several datasets in increasing order of complexity.
CLEVR-3D [33]. This recently proposed multi- camera variant of the CLEVR dataset, which is popular for evaluating object decomposition due to its simple structure and unambiguous objects. Each scene consists of 3-6 basic geometric shapes of 2 sizes and colors gray background. The dataset has ~-35k training and 100 test scenes, each with fixed views: the two target views are the default CLEVR input view rotated by 120 and 240", respectively.
randomly positioned
Figure 3: Example views of scenes from CLEVR-3D (left) and MSN-Easy (right),
MultiShapeNett-aay (MSN-Easy) [33]. This dataset is similar in structure to CLEVR-3D, however, 4 upright ShapeNet objects sampled from the chair, table and cabinet classes (for total of 122 objects) now replace the geometric solids. The dataset has 70k training and 100 test scenes.
MultiShapeNet-HHrk (MSN-Hard) [29]. MSN-Hard has been proposed as highly challenging dataset for novel view synthesis. In each scene, 16-32 ShapeNet objects are scattered in random orientations. Realistic backgrounds and HDR environment maps are sampled from a total set of 382 assets. The cameras are randomly scattered on a half-sphere around the scene with varying distance to the objects. It a highly demanding dataset due to its use of photo-realistic ray tracing [1 I], complex arrangements of tightly-packed objects of varying size, challenging backgrounds, and nontrivial camera poses including almost horizontal views of the scene. The dataset has 1M training scenes, each with 10 views, and we use test set of 1000 scenes. The ~51k unique ShapeNet objects are taken from all classes, and they are separated into a train and test split, such that the test set not only contains novel arrangements, but also novel objects. We re-generated the dataset as the original provided by Sajjadi et al. [29] does not include ground-truth instance labels necessary for our quantitative evaluation. We will make the dataset publicly available.

---

Table Quantitative results OSRT outperforms ObSuRF across all metrics and datasets with the exception of CLEVR-3D, where FG-ARI is nearly identical. On MSN- Hard, OSRT is able to encode
multiple input views to further improve object decomposition and reconstruction performance.
CLEVR-3D [33]
Masy [33]
MSN-HA Hard [29]
ObSuRF
OSRT (1)
ObSuRF
OSRT (1)
ObSuRF
OSRT (1)
OSRT (5)
PSNR
33.69
39.98
27.41
29.74
16.50
20.52
23.54
FG-ARI
0.978
0.976
0.940
0.954
0.280
0.619
0.812
4.1 Comparison with prior work
Tab. shows a comparison between OSRT and the strong ObSuRF [33] baseline. On the simpler datasets CLEVR-3D and MSN-Easy, ObSuRF produces reasonable reconstructions with accurate decomposition, though OSRT achieves significantly higher PSNR and similar FG-ARI.
On the more realistic MSN-Hard, ObSuRF achieves only low PSNR and FG-ARI. OSRT on the other hand still performs solidly on both metrics. Furthermore, while ObSuRF can only be conditioned on single image, OSRT is optionally able to ingest several. We report numbers for OSRT (5), our model with 5 input views, on the same dataset and see that it substantially improves both reconstruction and decomposition quality.
At the same time, OSRT renders novel views at 32.5 fps (frames per second), more than 3000 faster than ObSuRF which only achieves 0.01 fps, both measured speedup is the result of two multiplicative factors: S light field formulation is ~100 faster than volumetric rendering and the novel Slot Mixer is ~30 faster than the SB decoder here. Finally, OSRT does not need ground-truth depth information during training.
Nvidia V100 GPU. This
Fig. 4 shows qualitative results for the realistic MSN-Hard dataset. It is evident that ObSuRF has reached its limits, producing blurry images with suboptimal scene decompositions. OSRT on the other hand still performs solidly, producing much higher-quality images and decomposing scenes reasonably well from a single input image. With more images, renders become sharper and decompositions more accurate.
As the second relevant prior method, we have performed experiments with uORF [40]. Despite guidance from the authors, uORF failed to scale meaningfully the most interesting MSN-Hard dataset. This mainly resulted from a lack of model capacity. Additionally, the large number objects in this setting led to prohibitive memory requirements of the method, forcing us to lower model capacity even further, or to run the model with fewer slots to be able to fit it even on an Nvidia A100 GPU with 40 GB of VRAM. We describe this further with results in the appendix.
4.2 Ablations and model analysis
We conduct several studies into the behavior the proposed model including changes to architecture, training setup, or data distribution. Unless stated otherwise, all investigations in this section con- ducted on the MSN-Hard dataset with 5 input views. Further qualitative results for these experiments provided in the appendix.
Decoder architecture. As described in Sec. 2.4 and shown in Fig. 2, the novel Slot Mixer decoder differs significantly from the default SRT decoder [29], as well as from the Spatial Broadcast (SB) decoder that is commonly used in con- junction with Slot Attention. To show the strengths of the SM decoder, we com- pare with these alternative decoders switched in to the model, see Tab. 2.
Table 2: OSRT decoder variants Sloot Mixer combines SRT's efficiency with SB's object decomposition abilities. Results on simpler datasets are shown in Tab. 4.
MSN-Hard
PSNR
FG-ARI
FPS
SRT Decoder
24.40
0.330
40.98
Spatial Broadcast (SB)
23.35
0.801
1.39
Slot Mixer (SM)
23.54
0.812
32.47
7

---

Input
ObSuRF
OSRT (1)
OSRT (5)
Target
Qualitative results on MSN-Hard Comparison of our method with one and five input views with ObSuRF, which can only operate on single image. OSRT produces sharper renders with better scene decompositions.
Figure 4:
While the SRT decoder achieves slightly higher reconstruction quality due to the powerful transformer, result of the global information aggregation across
it does not yield useful object decompositions all slots, disincentivizing object separation the slots. is the key design choice Slot Mixer to only use transformer for slot mixing, but not for decoding the slots, that leads to good decomposition.
The SB decoder performs similarly Slot Mixer both terms of reconstruction and decomposition. However, due to the slot-wise decoding, it requires considerably more memory, which can often be prohibitive during training, and hamper scalability to more complex datasets large numbers of objects. For the same reason, it also requires significantly more compute at training and for inference.
Role of novel view synthesis. Our experiments have demonstrated OSRT's strengths in the default novel view synthesis (NVS) setup. We investigate the role of NVS scene decomposition, the complex MSN-Hard dataset.
To this end, surgically remove the NVS component from the method, while keeping all other factors unchanged, by training OSRT with the input views equaling the target views to be reconstructed. This effectively turns OSRT into multi-2D image auto-encoder, albeit with 3D-centric poses rather than pure 2D positional encoding for ray parametrization.
The resulting method achieves a much better PSNR of 28.14 14, 4.60 db higher than OSRT in the NVS setup (23.54). This is expected, as the model only needs reconstruct the target images rather than needing to generate arbitrary novel views of the scene. However, the model fails to decompose the scene into objects, only achieving an FG-ARI of 0. compared to OSRT's FG-ARI 0.812, demonstrating the advantage of using NVS auxiliary task for unsupervised scene decomposition.

---

Target
Prediction
Decomposition
GT Segments
Figure 5: OSRT trained on MSN-Hard (in color) is evaluated with grayscale input images at test time. We observe that the model generalizes remarkably well this out-of-distribttoon setting. This indicates that OSRT does not just rely on color for decomposition.
Robustness. We begin with a closer glance at the results obtained by OSRT (5) on MSN-Hard based on the number of objects in the scene. We find that the FG-ARI scores for scenes with the smallest (16) and largest (31) number of objects are within an acceptable range: 0.854 vs. 0.753,
To explore whether OSRT mainly relies on RGB color for scene decomposition, a grayscale version of the difficult test set of MSN-Hard. Note that the model was trained on RGB and has not encountered grayscale images all during training. We find that OSRT generalizes remarkably well to this out-of of-diiitibuttinn setting, still producing satisfactory reconstructions and scene decompositions that are almost up par with the colored test at FG-ARI of 0.780 vs. 0.812 on the default RGB images. Fig. 5 shows an example result for this experiment.
evaluate it on
We also consider to what extent OSRT is capable of leveraging additional context at test time in the form of additional input views. We train OSRT (3) with three input views and then evaluate it using three (PSNR: 22.75, FG-ARI: 0.794) and five input views (PSNR: 23.47, FG-ARI: 0.813). These results clearly indicate that additional input views can be used to boost performance at test-time.
Finally, we train OSRT in two setups inspired by SRT variants [29]: UpOSRT, trained without input view poses, and VOSRT, using volumetric rendering. We find that both of these achieve good reconstruction quality at PSNR's of 22.42 and 21.38 and meaningful scene decompositions at 0.798 and 0.767 FG-ARI, respectively.
Scene editing. We investigate OSRT's ability for simple scene editing. Fig. 6 (left) shows the a novel view rendered by OSRT. When the slot corresponding to the large object is removed from the SlotSR (center), the rendered image reveals previously occluded objects. We can go one step further by adding a slot from a different scene (right) to the SlotSR, leading to the object being rendered in place with correct occlusions.
Figure 6: Novel view (left) with slot removed (center) or a slot added from another scene (right).
3D consistency. An important question with regards to novel view synthesis is whether the resulting scene and object decomposition 3D-consistent. Prior work has demonstrated SRT's spatial consistency despite its light field formulation [29]. Here, we investigate OSRT 3D-consistency with regards to the learned decomposition.
To measure this quantitatively, we compute the ratio between the FG-ARI as reported previously- computed on all views together-and the 2D-FG-ARI which is the average of the FG-ARI scores for each individual target view. While FG-ARI takes into account 3D consistency, 2D-FG-ARI is unaffected by 3D inconsistencies such as permuting slot-assignments between views. By definition, 2D-FG-ARI is an upper bound on FG-ARI. Hence, an FG-ARI ratio of 1.0 indicates that the novel views perfectly 3D-consistent, while lower values indicate that some inconsistency took place.
We observe that our approach consistently achieves very high 3D consistency scene decompositions, with an FG-ARI ratio of 0.940 on the challenging MSN-Hard dataset. With 5 input views, OSRT achieves an even higher ratio of 0.987. Despite its volumetric parametrization, we find that ObSuRF often fails to produce consistent assignments with an FG-ARI ratio of only 0.707.

---

Table OSRT generalization on CLEVR-3D OSRT trained on scenes with 3-6 objects is tested on scenes with 3-6, and 7-10 objects, respectively. Both with learned and random slot initializations, OSRT generalizes reasonably to the presented out-of distribution (OOD) setting with more objects than during training. The slight drop in performance is mostly result of more pixels being covered by objects rather than the simpler background.
3-6 objects (IID)
PSNR
FG-ARI
7-10 objects (OOD)
PSNR
FG-ARI
Learned init.
40.84
0.996
Learned init.
33.03
0,955
Random init.
38.14
0.988
Random init.
33.36
0.968
cittrf-distribution generalization. We investigate the ability of the model to generalize to more objects at test time than were observed training time. To this end, trained OSRT either with randomly initialized slots, or with 11 slots using a learned initialization, on CLEVR-3D scenes containing up 6 objects. At test time, evaluate scenes containing 7-10 objects by using slots for both model variants.
The results are shown in Tab. 3. We find that generalization performance in terms of PSNR and FG-ARI is similarly good for both models, with slight advantage for the model variant with random slot initialization. In terms of in-distribution performance, learned initialization appears to have an advantage both metrics.
4.3 Limitations
In our experimental evaluation of OSRT, we came across the following two limitations that are worth highlighting: 1) while the object segmentation masks produced by OSRT often tightly enclose the underlying object, this is not always the case and we find that emergent masks can "bleed" into the background, and 2) OSRT can, for some architectural choices, fall into a failure mode which produces 3D-spatial Voronoi tessellation of the scene instead of clear object segmentation, resulting substantial drop FG-ARI.
While the alternative, yet much more inefficient, SB decoder does not seem to be affected by this, mask bleeding [9] effects and tesselation failure modes [18] are not uncommon in object-centric models. We show more examples and comparisons between the different architectures in the appendix.
Conclusion
We present Object Scene Representation Transformer (OSRT), an efficient and scalable architecture for unsupervised neural scene decomposition and rendering, By leveraging recent advances in object-centric and neural scene representation learning, OSRT enables decomposition of complex visual scenes, far beyond what existing methods able address. Moreover, its novel Slot Mixer decoder enables highly efficient novel view synthesis, making OSRT more than 3000 faster than prior works. Future work has the potential to elevate such methods beyond modeling static scenes, instead allowing for moving objects. We believe that our contributions will significantly support model design and scaling efforts for object-centric geometric scene understanding.
Acknowledgments
We thank Karl Stelzner and Hong-Xing Yu for their responsive feedback and guidance on the respective baselines, Mike Mozer for helpful feedback on the manuscript. Noha Radwan and Etienne Pot for help and guidance with datasets, and the anonymous reviewers for the helpful feedback.
10

---

References
[1] Titas Anciukevicius, Christoph H. Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint (rxXi::0044000644 2020.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoftrey E Hinton. Layer normalization, NeurlPS Deep Learning Symposium, 2016.
[3] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick. and Alexander Lerehner. Monet: Unsupervised scene decomposition and representation. arXiv preprint 1xkiv:9901.11.0. 2019.
[4] Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Inferno: Inferring object-centric 3d scene representa- tions without supervision, 2021. URL
2I1b.
[5] Chang Chen, Fei Deng, and Sungjin Ahn, Roots: Object-centric representation and rendering of scenes. J. Mach. Learn. Res., 22:259-15 1, 2021.
[6] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rna encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
[7] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 2018.
[8] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[9] Klaus Greff. Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
[10] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
[11] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermann, Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric; A Scalahle Dataset Generator. In CVPR, 2022.
[12] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge University Press, 2003.
[13] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 1985.
[14] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. 28, 2015.
[15] Wongbong Jang and Lourdes Agapito. CodeNeRF: Disentangled Neural Radiance Fields for Object Categories. In ICCV, 2021.
[16] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901-2910, 2017.
[17] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerehner, and Chris Burgess. Simone: View-invariant, temporally object representations via unsupervised vidco decomposition. In NewrlPS, volume 34, 2021.
[18] Laurynas Karazija, Iro Laina. and Christian Rupprecht. ClevrTex: A texture-rich benchmark for unsuper- vised multi-object segmentation. In NewrlPS Track on Datasets and Benchmarks, 2021.
[19] Thomas Kipf, Gamaleldin F Elsayed. Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In ICLR, 2022.
11

---

[20] Adam Kosiorek, Heiko Strathmann, Daniel Zoran. Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Rezende. NeRF-VAE: A Geometry Aware 3D Scene Generative Model. In ICML, 2021.
[21] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In ICLR, 2020.
[22] Li Liu, Wanli Ouyang, Xiaogang Wang. Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: survey. IJV,, 2020.
[23] Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[24] Ricardo Martin- Brualla, Noha Radwan, Mehdi M. Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[25] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NcRF; Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
[26] Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. In CVPR, 2021.
[27] William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336), 1971.
[28] Robin Rombach, Patrick Esser, and Björn Ommer. Geometry-frte view synthesis: Transformers and no 3d priors. In ICCV, 2021.
[29] Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth. Alexey Dosovitskiy, Jakob Uszkoreit. Thomas Funkhouser, and Andrea Taglinsacchi. Scene Representation Transformer: Geometry- Novel View Synthesis Through Set-Latent Scene Representations. In CVPR, 2022.
[30] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In ICLR, 2022.
[31] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene Representation Networks: Continuous AD-Structure-Aware Neural Scene Representations. In NeurlPS, 2019.
[32] Elizaheth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
[33] Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsuper- vised volume segmentation. arXiv preprint arXiv:2104.01148: 2021.
[34] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. Eurographics, 2022.
[35] Alex Trevithick and Bo Yang. GRF: Learning General Radiance Field for 3D Scene Representation and Rendering. In ICCV, 2021.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
[37] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander I Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vacs. arXiv preprint arXiv: I 1901.07017
2019.
[38] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In JCML, 2020.
[39] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixcINeRF: Neural Radiance Fields from One or Few Images. CVPR, 2021.
[40] Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section titled Limitations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [No] The new MSN-Hard dataset published on the website.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See experimental section and appendix.
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [No] The experiments require lot of compute and so use only single seed.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See experimental section and appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
The new MSN-hard dataset is published the website.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curatings [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We only use synthetically generated datasets of common household objects.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "lo6A3sO6LI", "reviewer": "Reviewer_VX3v", "summary": "This work tackles the task of view-synthesis and extends a prior work “Scene Representation Transformer” by introducing a slot-based bottleneck representation. This allows the model to learn disentangled representations using only the standard view-synthesis objectives, and the learned disentanglement shown to be more accurate the prior unsupervised methods. Moreover, this representation also enables object removal (although not more general editing e.g. translation/scaling that prior NeRF based unsupervised disentanglement methods would allow).\n\nThe proposed approach is evaluated using the CLEVR and Multi-ShapeNet. (MSN) datasets where it outperforms a prior work ObSURF in both, view synthesis and scene decomposition accuracy. However, the bottleneck representation does limit the model expressivity and the learned model is less accurate at view-synthesis than the base SRT model.", "questions": "N/A", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The overall approach is simple and intuitive. The key idea of adding slots after the SRT encoder makes sense and it would be valuable for the community to see the results.\n\n+ The ablations performed regarding the decoder architecture choice are informative. In particular, the proposed ‘slot mixer’ solution seems like a good compromise in retaining slot properties while being more efficient than the ‘broadcast’ approach in prior work.\n\n+ The reported improvements ObSuRF are clearly significant and highlight the benefits of adding slots in the SRT network i.e. ability to handle variable views and more accurate results.\n\n+ The ablation highlighting the role of view synthesis for learning accurate decomposition was very interesting, and shows necessity of using novel views for learning as opposed to mere auto-encoding objectives.\n\n\nWeaknesses:\n- While the ‘slot mixer’ vs ‘broadcast’ decoder is a useful contribution, the overall approach rather straight-forwardly combines ideas from prior works like UORF and ObSuRF with the SRT approach, and I feel the technical contribution here is a bit limited. That said, I still think this maybe a valuable (if obvious) combination for the community to see.\n\n- As a downside of using the proposed slot representation, the quality of the synthesized views does suffer. While this is only highlighted in Table 2 on one dataset, it would be helpful. to see similar results across all datasets.\n\n- (minor point) I feel the claim of “3000x faster” rendering is a bit misleading as this mostly comes from the speedup obtained by SRT over volume-rendering based methods, and isn’t really a contribution of this work (the ‘slot mixer’ vs ‘broadcast’ speedup could be attributed to this work, but that is a much smaller factor).\n——\n\nOverall, I think this is simple but obvious paper. On the one hand, I am not sure this work makes any remarkable technical contributions or empirical findings. But on the other hand, it represents a well executed and sensible combination which the community would benefit from seeing, and I would therefore lean towards accepting it.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WiNyOfOwdLN", "reviewer": "Reviewer_3uqV", "summary": "- The paper addresses the problem of unsupervised decomposition of scenes into a discrete set of objects. The idea is that novel view synthesis requires reasoning about the geometry which should also be helpful for discovering objects without additional supervision. A novel transformer-based approach for object centric 3D representation learning is introduced which aims to overcome limitations of previous methods regarding generalization to complex scenes due to high computational costs.\n- The method uses a slot-attention mechanism in combination with a light-field rendering formulation using a transformer conditioned on a viewpoint-dependent latent representation derived from these slots. To avoid high computational costs, an early fusion scheme is presented that mixes slots early and thus has to run the decoder only once and not per slot.\n- Compared to [27], the proposed approach does not require depth supervision for efficiency as it relies on an SRT [23] based decoder for efficient rendering instead of volumetric rendering approaches.", "questions": "- Fig. 3 in the supplementary seems to indicate that the Spatial Broadcast (SB) decoder suffers less from oversegmenting. Qualitatively it also seems to be more accurate around object boundaries but the quantitative results in Tab. 3 say the opposite. What could explain this discrepancy and could the proposed Slot Mixer (SM) decoder benefit from an explicit background modeling to avoid these problems?", "limitations": "Limitations and potential negative societal impact have been addressed adequately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - The introduction provides a good motivation towards the goal of learning about compositions and the geometry of scenes solely from observations. The related works section provides good context regarding neural rendering, object centric learning and specific differences to closest related works on object centric 3d learning.\n    - Moving unsupervised 3d scene decomposition towards more complex and realistic scenes is an important goal with many applications. The proposed method demonstrates good performance on the more complex MSN-Hard dataset and could serve as a valuable baseline, especially in combination with the dataset's ground-truth instance labels.\n    - The proposed approach is simple and reasonable. Its evaluation supports the stated claims regarding the presented model.\n    - The experimental comparison to [27] (as well as the description of the attempts to train [5] in the supplementary) convincingly demonstrate the advantages of the proposed model in terms of accuracy and runtime, especially when going to more complex datasets.\n    - The design of the ablations allows for a good analysis of the different components involved in the approach. The decoder analysis clearly demonstrates how the SM decoder improves decomposition over an SRT decoder [23] and runtime over an SB decoder. The experiments on the role of novel view synthesis for decomposition validate the hypothesis that novel view synthesis indeed helps object decomposition. Evaluating whether the approach is mainly segmenting by color cues is a good test for generalization capabilities of the approach. Scene editing experiments give a nice qualitative indication that slots indeed represent meaningful objects.\n- Weaknesses\n    - The novelty of the proposed method is limited, as it is a rather straightforward adaptation of slot attention [17] to SRT [23] with an early fusion scheme for faster decoding.\n    - Qualitatively (Fig. 3 of the supplementary), the SB decoder still looks better than the proposed SM decoder, although this comes at a large computational cost. The quantitative results suggest that the SM decoder performs better, which makes me question the metric a bit.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K59pMKllt7S", "reviewer": "Reviewer_HAVj", "summary": "The paper presents a method to infer an object-centric scene representation aware of 3D geometry from multiple views of a scene. Similar to other works, it uses slot attention to segment scenes into slots, and it uses a ray-casting approach to rendering 2D views of the represented scene. However, the authors propose a novel decoder, based on Scene Representation Transformers (SRT), that only requires a single evaluation per ray query regardless of the number of objects in the representation. This makes the model faster than previous alternatives, while also achieving good quantitative metrics on different datasets.", "questions": "- How is the background handled?\n- Differently from Slot Attention, it seems that the initial slot representation is learned. How does that affect the model performance and how does it affect its generalization capabilities to an a different number of objects from training/testing?\n- How does the model compare to UORF quantitatively on the simpler datasets proposed in that paper?\n- The segmentations compared on the FG-ARI metric do not take into account that some object segmentations might be covering the background. How do the metrics change when using the ARI metric", "limitations": "The authors mention limitations of their work and ethical considerations. The limitations of their work could be better analyzed by conducting some of the experiments suggested in the Questions section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n+ Clear presentation:\n\nThe problem, method and results are presented clearly and it is easy to follow the paper.\n\n+ Fast approach to decoding 3D object-centric representations:\n\nThe model has much lower computational costs per iteration given that the number of evaluations per ray do not scale linearly with the number of objects. This makes the model fast and more scalable than other alternatives.\n \n+ Good performance compared to ObjSURF:\n\nThe model outperforms ObjSURF both quantitatively and qualitatively on different datasets.\n\n\n**Weaknesses:**\n\n- Missing references:\n\nThe paper is missing numerous relevant references. Specifically, it is missing the whole body of work of Sungjin Ahn's team, with ROOTS[1] being directly related to this paper, and relevant work by others [2,3].\n\n- Evaluation could be improved:\n\nWhile the paper compares in three datasets of increasing difficulty, the comparison is mostly to ObjSURF. The paper only compares to UORF on the harder dataset, in which it does not work well due to the high number of objects and computational costs associated. While this is a good comparison to highlight the better scaling capabilities of the proposed model (and this is an important point), it does not resolve the issue of how this model compares to competing approaches when they can actually compete. An easy way to solve this would be to train UORF and other previous methods on the simpler datasets, or compare on one of the proposed datasets in the UORF paper. In fact, there are experiments with different backgrounds and more complex objects in that paper, that are similar to the proposed datasets in this paper.\nIt would also be interesting to understand the quality of the segmentations by reporting regular ARI metrics, instead of just FG-ARI metrics, as these can be misleading when objects are not segmented tightly.\n\n\n**Detailed comments:**\n\nOverall this is a promising submission. While the idea of having an object-centric SRT model is relatively straightforward, the proposed decoder is novel and has clear advantages in terms of computational costs while still being interpretable. However, the paper could be improved with a more thorough literature review and a better comparison to previous approaches based on this literature review. There are a few questions to be answered, but overall I believe this can be a good submission with some modifications. I am currently arguing for its acceptance, and I would be happy to increase my score if the authors address my comments.\n\n**References:**\n[1] Chen, Chang, Fei Deng, and Sungjin Ahn. \"ROOTS: Object-Centric Representation and Rendering of 3D Scenes.\" J. Mach. Learn. Res. 22 (2021): 259-1.\n[2] Castrejon, Lluis, Nicolas Ballas, and Aaron Courville. \"INFERNO: Inferring Object-Centric 3D Scene Representations without Supervision.\" (2021).\n[3] Anciukevicius, Titas, Christoph H. Lampert, and Paul Henderson. \"Object-centric image generation with factored depths, locations, and appearances.\" arXiv preprint arXiv:2004.00642 (2020).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AH0VNcK4abc", "reviewer": "Reviewer_jvub", "summary": "The paper proposes Object Scene Representation Transformer, a framework for learning scene representation. The framework combines scene representation with the Slot Attention that constructs a bottleneck such that object-centric representation naturally emerges from learning. The proposed method is fast and achieves good performance.", "questions": "See above.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Let me start by honestly stating that my research background is in representation learning not in 3D understanding or rendering, therefore my comments should not serve as the key evidence for deciding the outcome of this submission.\n\nThe paper is well written and easy to understand even for someone who doesn’t work in the field. The technical details are sound and the claims are fair. The experimental evaluation seems comprehensive. The performance looks good.\n\nAs my background is in representation learning, I’d like to raise a few questions on its application in real-world and complex scenes. Object centric representation is mostly learned through adding inductive biases into a system, such as the Slot Attention adopted in the paper. A bottleneck stage can force a neural network to group pixels into a compact representation, and object *naturally emerges*. However, in my opinion and from my research experience, nothing naturally emerges in a neural network – it does what it does because we, the creators, told it so. In other words, we achieve this grouping by injecting inductive biases into the architecture.\n\nI have always been cautious about injecting inductive biases because doing so mostly helps *low data, low compute* regime, but many times harms a model’s capacity to scale to more data and more compute. Take the example of ViTs vs. CNNs – ViT models have much less inductive bias but they need more data to train and start outperforming CNNs (by a significant margin) as the model sizes grow. Inductive bias is a double-edged sword and any practice of adding inductive bias should be well motivated and examined with extreme caution. \n\nBack to the scope of this submission – I found that the datasets used here are rendered rather than from real-world. Naturally in these datasets object properties, such as numbers, lighting, materials, are randomized in a limited range. This setting seems to match the concerning low data, low compute criteria, therefore more inductive biases, especially dataset engineered ones, will unsurprisingly shine among baselines. It may not be the case for real-world data in which case the scalability and capacity of a model can truly be tested.\n\nI’d like to note the above comments are made without a comprehensive investigation of common practice in similar papers and domains. I plan to take all reviewers’ comments into consideration, and after reviewers’ discussion a further recommendation will be made.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept