Scalable 3D Object-centric Learning
Authors: Tianyu Wang
miaomiao Liu
Kee Siong Ng
Venue: Submitted to ICLR 2023 (2023)

Abstract: We tackle the task of unsupervised 3D object-centric representation learning on scenes of potentially unbounded scale. 
  Existing approaches to object-centric representation learning exhibit significant limitations in achieving scalable inference due to their dependencies on a fixed global coordinate system. 
  In contrast, we propose to learn view-invariant 3D object representations in localized object coordinate systems. 
  To this end, we estimate the object pose and appearance representation separately and explicitly project object representations across views. 
  We adopt amortized variational inference to process sequential input and update object representations online. 
  To scale up our model to scenes with an arbitrary number of objects, we further introduce a Cognitive Map that allows the registration and querying of objects on a global map. 
  We employ the object-centric neural radiance field (NeRF) as our 3D scene representation, which is jointly inferred by our unsupervised object-centric learning framework. 
  Experimental results demonstrate that our method can infer and maintain object-centric representations of unbounded 3D scenes. 
  Further combined with a per-object NeRF finetuning process, our model can achieve scalable high-quality object-aware scene reconstruction.

Under review as a conference paper at ICLR 2023
SCALABLE 3D OBJECT-CENTRIC LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle the task of unsupervised 3D object-centric representation learning on scenes of potentially unbounded scale. Existing approaches to object-centric repre- sentation learning exhibit significant limitations in achieving scalable inference due to their dependencies on fixed global coordinate system. In contrast, we propose learn view-invariant 3D object representations in localized object coordinate systems. To this end, we estimate the object pose and appearance representation separately and explicitly project object representations across views. We adopt amortized variational inference to process sequential input and update object rep- resentations online. To scale up our model to scenes with an arbitrary number of objects, we further introduce a Cognitive Map that allows the registration and querying of objects on a global map. We employ the object-centric neural radiance field (NeRF) our 3D scene representation, which is jointly inferred by unsu- pervised object-centric learning framework. Experimental results demonstrate that our method can infer and maintain object-centric representations of unbounded 3D scenes. Further combined with a per-object NeRF finetuning process, our model can achieve scalable high-quality object-aware scene reconstruction.
INTRODUCTION
The ability to understand 3D surroundings in an object-centric way is crucial for Al agents to perform range of tasks including relational reasoning (Chang et al. 2017) and reinforcement learning (Diuk et al., 2008). In recent years, 2D and 3D unsupervised object-centric learning have attracted increasing attention in the field. Compared with 2D object-centric learning methods (Eslami et al.)
2016; Lin et al. 2020 [Burgess et al.) 2019: Crawford & Pineau 2019, Locatello et al.) 2020) that focus on decomposing 2D images into objects, 3D learning methods aim to recover the complete
3D scene structures in an object-centric way using RGB or RGBD observations (Li et al. 2020
Stelzner al. 2021 Chen et al. 2021 Henderson & Lampert 2020). A key limitation of existing 3D methods is that they can only handle scenes with a scale that can fit into the field of view (FOV) of a fixed number of cameras et al., 2020; Stelzner et al. 2021 Chen et al. 2021). where the camera coordinates are specified within global coordinate system. The trained inference models are thus highly dependent on the chosen global coordinate system (Li et al., 2020 Chen et al. 2021 Kabra et al. 202T Eslami et al.] 2018 Henderson & Lampert, 2020). and cannot generalize beyond the scale of the training sets. All of these limits the applicability existing 3D methods to real-world problems
simulated reinforcement learning environments (Beattic et al. (2166), where scenes with large or even unbounded scales are routinely encountered.
In this paper, we propose Scalable Online Object Centric network in 3D (S0OC3D). S0OC3D addresses the problem of scalability by inferring object poses and view-invariant object representations in localized object coordinate systems from RGBD data. To handle sequential data for large-scale scenes, we exploit amortized variational inference for online inference. Inferred object poses allow object representations be explicitly projected across views with preserved identities. To keep track of all the detected objects throughout the online update, we introduce a highly scalable external memory mechanism named Cognitive Map which can be used to dynamically register and query detected object representations. This memory mechanism further removes constraint in existing works (Henderson & Lamperi. 2020 |Burgess et al.) 2019, Locatello et al.) 20200, EEgelcke al.. 2020a Yu et al. 2022) whereby the maximum number of objects allowed in each scene is capped.
'The term cognitive map is borrowed from cognitive psychology studies on mental representations of the spatial surroundings in animal, and human brain (Kitchin 1994).

---

Under review as a conference paper at ICLR 2023
We adopt the 3D object-aware Neural Radiance field (NeRF) to decode such representations to 3D geometries for training. While per-scene NeRF with direct SGD optimization can capture detailed 3D scenes (Mildenhall et al. 2021 Zhang al.; 2022. Tancik et al. 2022), the reconstruction quality of such unsupervised object-centric NeRF learning methods commonly falls short of the per-scene NeRF approaches as the introduced information bottlenecks filter out High-frequence information (Engelcke et al., 2220b)). To narrow the gap, we introduce the per-object NeRF finetuning process to improve the reconstruction quality while preserving the objects' identities.
Our contributions are summarised as follows. i) We propose, to the best of our knowledge, the first unbounded scalable generative- cooe-mooolllaleeeeeeeeeee based unsupervised 3D object-centric learning framework. ii) We learn the explicit object poses and view-invarian object representations separately via the amortized variational inference framework to achieve scalable online updating- iii) To store a potentially unbounded number of objects detected for scalable inference, we introduce Cognitive Map separating object representations management from the inference process. iv) We demonstrate that the reconstruction quality can be further improved via our per-object NeRF finetuning process with preserved objects' identities.
Asz
query
register
Zap Aes
Amorii and Veriational Inference
Aaget
Aspe1
Azzui
componentss
Pures
Aze
Agr
Aarei
Decoder
RefinoNet
Agp
Agges
11
Razes
Ag
Ax
Ag
Ax
Aazes
Decoder
Aare1
2
Amartbec librence
compasition
Figure Left: The interaction between the inference pipeline and the cognitive map for scene updating (top) and novel view synthesis (bottom). Previously detected objects are registered in the cognitive map. When new view arrives, the representations of all objects present the view are retrieved. } are the object representations and X is the scene layout representation. If the number of objects existing the current view is less than a pre-defined value K, add prior representations (greyed A₂). We update {A:} } to integrate new information with the amortized variational inference process and register them back into the cognitive map. For novel view synthesis, the retrieved representations are decoded into NeRF components. By components composition, we synthesize RGB image, depth, segmentation and uncertainty map. Right: L-iteration amortized variational inference process. In each iteration, the set of input representations is decoded into NeRF components. We evaluate the likelihood of observation under the composed NeRF. The refinement network takes the decoded NeRF, observation and likelihood to update the object-centric representation.
RELATED WORK
Unsupervised Generative Model-based 2D Object-centric Learning. 2D object-centric learning aims to group pixels covering the same object under the same label and at the same time produce neural representation of each discovered object. At the core of those methods is the spatial mixture model formulation that frames object-centric learning as a latent variable inference problem (Eslami
al. 2016; Greff et al.
2019
2017p. To handle observations with a high object density. a branch
of works (Eslami et al.
2016
Crawford & Pineau, 2019. Lin et al., 2020) infers latent variables
for local regions of each
observation. Pipelines equipped with iterative refinement modules
input view. Particularly, IODINE (Greff et al. 2019) employs amortized variational inference (Marino) al. 2018) that can process sequential data. However, the aforementioned methods do not infer 3D> structures. Object latents are discarded once out of view.
(Greff et 2017 Locatello et al. 2020) refine the latent variable iteratively conditioned
Unsupervised Generative Model-based 3D Object-centric Learning. 3D-awere methods not only try to factorize observations in an object-centric manner but also infer the 3D spatial structure of scenes, which can be examined by the means of novel views synthesis. Similar to its 2D counterpart,
2

---

Under review as a conference paper at ICLR 2023
3D object-centric representation learning approaches also adopt the spatial mixture model formulation. ObSuRF (Stelzner et al. (02T) and uORF (Yu et al.] (022) introduce neural radiance fields (NeRF) into the object-centric learning setting. Smith et al. (2022) propose to use object light field to avoid dense sampling along rays during rendering. As an attempt to model scenes on a larger scale, O3V (Henderson & Lampert, 2020 and SIMONe (Kabra al., (021) infer object-centric representation from a video sequence. However, both O3V and SIMONe adopt non-incremental method and process entire video sequences before generating scene representations, MulMON (Li et al. (020) adopts amortized variational inference framework for 3D object-centric learning that allows object latents to be updated by new views in an online fashion.
The methods mentioned above work well on scenes with bounded sizes that can fit into the FOV of camera cones. One limiting factor of scaling up to larger scenes is that they take camera poses and locations specified in a pre-defined global coordinate system as input to the network. Thus, during test time, they cannot handle scenes larger than the ones in training sets. Our work infers view-invariant object representations that can scale up to unboundedly large scenes.
and Scalable NeRE Object-compositional NeRF has been studied recently to learn the 3D representation for each object and the scene for image synthesis. In particular, Yang ct al. (202I) introduced a two-pathway framework to model the foreground objects and the scene branch, with known coarse object instance masks. Such method cannot be directly applied to the unsupervised scenario. To handle large scenes, block-wise NeRF has been proposed recently
et al. 2022 Tancik al., 2022). They can either generalize to large scene (Zhang al.; 2022] by taking multiple images as input or train scene-specific block-wise NeRFs for large scene fast rendering (Zhang et al. 2022). However, both Zhang et al. (2022) and Tancik et al. (2022) are not object-centric. In this paper, we aim to combine merits from all sides and learn scalable object-centric NeRF scene representations.
3 METHOD
With the assumption that the scene static and there can be at most K objects in each view (but unlimited in the entire scene), SOOC3D aims to achieve object-centric 3D scene understanding in unbounded scale during test time from sequential RGBD inputs. This goal requires the inference process to be online and view-invariang recurrent variational proposal distribution (Sec. B.T. SOOC3D ensures a view-invariant object representation by decoding NeRF under the object coordinate system. The refinement network takes both view-dependent observation and view-invariant object representation to update the object latent variables (Sec. 3.2). Notably, the set of objects in view may change when switch from one camera to another. To keep track of all object representations and only expose those that are in view to the inference process, we introduce Cognitive Map for scalable learning (Sec. 3.2) The illustration of the inference pipeline is shown in Fig. []. Finally, we show that object reconstruction quality can be improved with preserved identities through the per-object NeRF finetuning process with an
In SOOC3D, the online inference is facilitated by the
object-centric initialization (Sec. 3.2
3.1 GENERATIVE MODEL FORMULATION AND THE OPTIMIZATION OBJECTIVE
Generative Model. At any time I, we have access to set of camera views V {ur. {ur..... ... U/ specified by their extrinsic parameters. Each camera captures a RGB Cv, observation, and depth observation de,, We define o,, {Co,, de,, } and O, {ou,..... Ov, Below we hide the subscript of camera pose in equations that hold for all camera poses. We assume there are at most K objects obj
in one view. The K object latent representations form a set 2 = {>ab] (1)
(K())) where
(.(.) returns the global index of each object latent given its ine index. The set of all object latent variables at time t is 2 = {21 .......
obj = UvEv. 2
The latent variable of an object is specified 2; شا² {auhhere, 240hat ..res). }, where 2 ztheee R3 encodes object location on xz-plane and rotation around y-axis, 2. z E RD encodes object appearance. _pres E {0, 1} is a binary variable with z press 0 indicating that object does not exist and 2 = 1 otherwise. We further define 250one E R as the latent variable to model the scene layout under camera We assume that for each view, the scene layout component always exists and
3

---

Under review as a conference paper at ICLR 2023
the location of scene layout component is always the center of the camera view. We denote 2, as
sonne, The complete data likelihood function defined as: 2 2g
E, (()o)
(I)
Amortized Variational Inference. The exact object latent posterior p(a,.
ok Or V.) is intractable.
Thus, we resort to the amortized variational inference (Greff et al. 2019 Li et al. 2020
Emami et al.
2021
We approximate the true posteriors by proposal distribution
obj Oc, Vi) q(=,,t
obj q( Our Ub, 2,111 (121))9(((,1110000 V. V
lo
obj
2003
with obj JOo, Vo) being the variational prior. The recursive nature of the variational posterior enables online inference with a constant memory footprint. When object i not visible in view
et, we define (( [0n; Ut, =1, to be the Dirac delta function any (z),e) The posterior will
remain unchanged as q(2,,
abj 10,, V:) obj 1(2,,--100-11, Vi-1).
As an approximation, we further assume that 2 2.hhere, 22 22 are independent of each other conditioned on Or, V. We parameterize ((z(hhrr)) and q(2,hhat) as isotropic Gaussian with X Awhere {u o.ohere) and Awshat {nrthat o (ohatt. {(=)Te)) takes the form of Bernoulli distribution and the Xpriee is the logit. Following the amortized variational inference, for each input view Us, we E 10,1, [.} iteration,
update our latent for iterations. At the E
obj. Fie
9x1, (Zi,t Ovvr ut, Zi,t
(o obj.l-1
(3)
Xr, = X1,7 fo( objd-1
Oue ve, a)
with Exp, = - fo
the refinement network and a is collection of auxiliary input. For each view, we compute the KL-Divergence C/: = Σ/= PKIlyx [19xg,]) The iteration is executed in parallel for all object latent distributions that detected in the current view.
During training. for each scene, addition to T input views Vr, we also sample a set of query views Q. The final Evidence Lower Bound (ELBO) is defined
1 T
1 Σ DEQ
1
L
ggg [logp(ov. 12,,] +
Bus [og
cki
(5)
Q
T
We adopt the same pre-ray likelihood function as Stelzner et al. 2021) to compute p(o, Z.). For the completeness of the paper, we detail the likelihood function computation in Appendix
3.2 MODEL IMPLEMENTATION
Coordinate System Transformation. The iew-invariant inference depends heavily on the coordi- nate system transformation between camera coordinate systems and the object coordinate systems. For each step, our model expects an RGBD observation and latent variables Z from the previous step. As detailed in Appendix B, along each ray we get the surface sample at the observed depth d, and one air sample with a depth less than d. We specify each sample æ in the camera coordinate system.
Each where is interpreted as object pose in the current camera coordinate system. With the object (k) poses specified, for each object indexed by k, we can build projection matrix IT( ((==hhe (22eher)) E (3)
and map each point x the object local coordinate system by 24 (heeee x for decoding. The global coordinate system only used for object registration and query.
NeRF decoding, Following common practice, we transform the coordinates of samples into harmonic representations (Mildenhall et al. 2021 Conditioned on whatt or 2 2:c0me NeRF decoder o.(-) (k).
assigns each point a raw density 0% ( ōo(*k, what ) E [0, 1] and a RGB color. While the K object components share the same NeRF decoder, the scene layout is decoded via its own decoder.
4

---

Under review as a conference paper at ICLR 2023
To accurately predict object location, we introduce inductive bias in the form of Gaussian weighting.
To be more precise, we compute the weighted density log 0% (*x) = log (xx) + log "y (G (xx)) + log -eee where wy(-) is a zero centered gaussian function and SG is the stop gradient operation. By adding log (x(x)) SG (log wi (I+)), we encourage the zwherre to be set at (1)
the object center with the value of the weighted density unchanged. Weighted by (xx (k)) non-existent
zrres
components are turned off. We then compute the normalized density as (x*x Σi= d.(z,) Note that >x oR (Tx) E [0, I] allowing us to represent concrete object or void space. The final NeRF
dg (mx))
density at point is given by (x) = Omax Eb=o (x( with maa being the maximum NeRF
density of our choice.
We do not down- down-weight air sample densities by w and 2tre force the model to learn the scene voidness at all ranges. At this point, the decodding/reconstruction is complete and we can compute the reconstruction likelihood.
Refinement Network. Following the amortized variational inference literature (Greff et al. 2019 Li et al., 2020), refinement network takes as input latent variables from the previous step and a set of auxiliary data and outputs the updated proposal distributions (Eq. 4 The auxiliary data include observation, previous reconstruction and observation likelihood. For each input view, the above reconstruct-and-rede our objective function enforces the refinement process to preserve object identities across views without any hard-coded heuristics (detailed in Appendix B Detailed network structure, auxiliary input specifications and algorithmic summary are presented in Appendices FJ [A]
process is executed L times, in parallel for all components. Notably,
Cognitive Map. A cognitive map maintains all discovered objects with list where the 8th entry stores A, for object and interacts with the inference model through two functions:
Query: Given camera specified in the global coordinate system, we retrieve {(i, for K objects whose locations are inside the field of view for camera v, from the cognitive map. To this end, according to the extrinsic parameters of v. μ wheree for all objects in the cognitive map projected from the global coordinate system into the camera coordinate system of v while other parameters are kept fixed. If less than K objects are located in the current view, priors of latent variables are filled in with pseudo index = 11
Registration: Given camera v, the set {(i, Ax) ie (1)...)) (1).... (K) can be registered into cognitive map. Objects with (==Pre = 1) < 0.5, will be discarded since they are deemed non-existent. As the reverse process of a query, all μ where will then be projected into the global coordinate system. Then, newly discovered objects (identified by i -1) are appended to the list and the corresponding entry indices become their global object id.
Per-Object NeRF Finetuning. To achieve higher reconstruction quality, we associate each object representation with a per-object NeRF (no weight sharing) in the cognitive map. We initialize the per-object NeRF decoder that of the SOOC3D NeRF and then finetune by maximizing the input view observation likelihood. Such non-weight sharing and training strategy for per-object NeRF can improve the reconstruction quality and preserve object identity naturally. For implementation. we fix uheeee binarize grres with a threshold of 0.5 and treat view 2 what as part of the NeRF parameters.
4 EXPERIMENTS
Dataset. Datasets adopted in object-centric learning literature commonly focus on small scenes that can fit into camera FOV (Eslami et al., 2018 Johnson et al.) [017 Engelcke et al. 202 Yu et al.) 2022). To evaluate the scalability of our approach, we construct a large-scale dataset termed Unity dataset mimicking the object room dataset (Eslami et al., 2018). The dataset contains scenes of three different scales termed as small (s), medium (m) and large (l). To assess the performance of our model when facing non-trivial geometries, we also build a Blender dataset where objects are randomly selected from pool of indoor furniture. See Appendix for dataset generation details.
Metric. We compute the mean- mean-intersenn intersection pnserseetioo-ovve-uniioo over-union (mloU) score between the ground truth mask and the mask inferred from our NeRF representation for scene segmentation and per-pixel root-mean- square-error (RMSE) for the rendered RGB image and the depth for the scene reconstruction. We mask out all pixels with depth values larger than the clipping plane. We compute both mloU and
5

---

Under review as a conference paper at ICLR 2023
RMSE for both input (I) and query (Q) views. We also evaluate the location prediction performance by measuring the average distance between the predicted object locations and the ground truth for each scene. See Appendix for rendering equations of depth, segmentation masks, and uncertainty maps. Unless stated otherwise, the visualizations below are without per-object NeRF finetuning.
Baseline. We compare our method with MulMON et. 2020), the state-of-the-an art multi-view 3D scene object-centric learning method with online inference ability. We adopt their official implementation and additionally add ground truth depth as input. We additionally discuss ObSuRF Stelzner et al. (2021), a single view inference model, in Appendix to provide more insights.
4.1 SCALABLE OBJECT CENTRIC LEARNING
Test time scalability is crucial for deploying agents in an open environment. Below we identify three types of scalability desired for an object-centric learning model. Our experiments show that our model demonstrates strong test time scalability across all three types.
Scene Scale. In practice, the training scenes are normally of bounded scale but the deploying scenes are of varying sizes, Below, we test the generalization of our approach w.r.t a different scene scale. Specifically, we train the baseline and our model on both small (MulMON -mall, Ours_small and medium (MuIMON medium, Ours medium) scenes and test them on different scene scales. For the baseline, set the number of mixing components M 8, 12, and 25 for small, medium and large scale scenes respectively, during both training and testing. For our model, we fix K to 7 for all scene scales. The quantitative results reported in Table T The results of our approach are obtained as the average of 5 runs with at most #0.01, +0.02, +0.02 and =0.02 variation for mIoU, RGB-RMSIE depth-RMSE and coordinate L-2 error respectively. MulMON achieves 0.612 mloU when trained and tested on small scenes. This setup aligns with their assumption that all objects appear in all views. However, evaluated on medium and large-scale scenes, its performance drops significantly below 0.2 in mloU. The performance of MulMON is not improved after training on the medium scene.
Table 1: Quantitative results on scene segmentation and reconstruction.
mloU
RGB RMSE
depth RMSE J
caord.
m
m
m
MulMON_smail
0.198
0.158
0.055
0.167
0.178
N/A
0.599
0.192
0.152
0.056
0.171
0.186
VA
NAA
0.371
0.225
0.141
0.102
0.141
0.144
NA
N/A
MalMON
0.365
0.221
0.103
0.159
N/A
N/A
Ours Ours_small
0.763
0.721
0.694
0.074
0.107
0.141
0.516
0.680
0.793
N/A
Q
0.761
0.713
0.690
0.033
0.109
0.149
0.517
0.691
0.805
0.068
0.170
0.192
0.710
0.756
0.761
0,075
0.098
0.091
0.617
0.650
0.634
N/A
Oedsume
0,703
0.751
0.757
0.073
0.099
0.092
0.619
0.652
0.640
0.099
0,117 0.111
Oissttineeune
0.903
0.897
0.910
0.016
0.018
0.021
0.472
0.620
0.615
N/
0.892
0.874
0.889
0.019
0.023
0.025
0.498
0.630
0.624
N/A
Ground Truth
MuLMON
Figure Qualitative comparison on query view synthesis large scenes.
By contrast, our model, being agnostic to any global coordinate system, can scale to large scenes with a 0.07 mIoU performance drop when it only trained on small scenes. After we train our model on medium scenes, the performance in both medium and large scenes improves (see Fig. 2 for qualitative comparison). The similar performances on the input and query views demonstrate that our model learned the view-invariant features resulting in robust rendering from all views. As shown in the last row of Table after per-object finetuning, on the unity dataset, both instance mask prediction and appearance reconstruction improved significantly (qualitative results are shown in Appendix H)

---

Under review as a conference paper at ICLR 2023
Cog. Map
Ground Truth
Reconstruction
Figure 3: Visualization of an online inference process from top bottom. Each row corresponds to one inference step. The left column shows the evolution the cognitive map. The camera pose of each step is marked by blue triangle and the camera cone (visible area) is highlighted. Each object latent registered in the cognitive map is marked with an x and greyed out if outside of view.
In Fig. we visualize a 6-step online inference process. For each step, our model discovers new objects, registers them to the cognitive map and at the same time updates the latent variables. With cognitive map as external memory, the inference process can be scaled to arbitrary scales.
The Number of Object in View. To demonstrate the scalability w.r.t the number of objects in the scene, we generate a set of 100 testing scenes, each of which consists of 10 views. Fach view can capture 11 objects. We set the K be 11 and test our pre-trained model without any additional training. We then report that our model achieves on average 0. 702 mloU, and 0. 0.092 for RGB-RMSBD Although it has never been trained on such dense views, our model can generalize well dense 3D scenes without performance drops. Qualitative results on scene decomposition are shown in Fig. 4 Our method can decompose the images with dense objects and render each object with high quality.
Figure 4: Demonstration of the model scalability at test time the number of objects per view. In rows and 3, from left right are ground truth RGB, depth, instance mask, rendered RGB, rendered depth, rendered mask and uncertainty map. Rows 2 and 4 show the rendered individual objects.
Number of Updates. practice, objects can be observed and updated arbitrarily frequently and the inferred scene representations should remain stable. However, during training, it common that each
7

---

Under review as a conference paper at ICLR 2023
object latent can only be updated fixed maximum number of times. In our experiments, object latent can be updated times at most during training.
To evaluate model's scalability object latent updates without additional training, sample input views and repeat the inference process multiple times. After each update, we compute the induced KL Divergence and the RGB-RMS for the rendered image. As shown in Fig. | model shows test-time scalability to around 10 updates. To achieve stronger scalability, we finetune our model by traversing each scene 5 times consecutively. Similar to the truncated backpropagation through time (TBPTT), after each traverse, we update the model parameters and keep the contents of the cognitive map for the next traverse. result, the object representation remains stable under more than 30 updates during testing time. Fig. 6 visualizes the appearance changes under updates.
200
TBFTT
wi TIPTT
130
we TIPTT
100
0.00
-ppaac
Figure 5: The per-update KL (left) and per-update RGB RMSE (right) with and without finetuning.
Ground Truth
nd update
2 update
4
update
update
update
10
update
12
update
Ground Truth
1s update
2 update
10 update
15%
update
20
update
25
updlte
30
updlate
Figure 6: Visualization of scene changes under update before (top two sequences) and after (bottom two sequences) the TBPTT finetuning. Due to object blocking views, the first update may result in high uncertainty. Qualitatively, the appearance objects refined and remains stable after.
4.2 SCENE EDITING
In previous works, while object removal creation can be achieved by adding or deleting object latent variables, moving objects commonly relies on latent variable traversal (Kabra et al., [021] Burgess al. 2019 Greff et al., 2019 Li et al. 2020). For our model, thanks the explicit object pose estimation, scene editing can be achieved with direct modification of the content of the cognitive map in a view-independent way. We use our model to infer the latents of a random small scene and save the resulting Cognitive map. In Fig. 7 we modify the registered coordinates of the latents in the Cognitive map by directly setting them to some designated coordinates to create different object arrangements. In Fig. 8 we traverse whaat of the left-most object dimension the modified object and inspect the changes in the object appearance manually. We identified three feature dimensions with highly interpretable meanings, i.e. color, shape and size.
time, render
8

---

Under review as a conference paper at ICLR 2023
4.3 VISUALLY CHALLENGING SCENES AND PER- CER-OBJECT FINETUNING
We test our model on the Blender dataset to evaluate the capability of handling more realistic scenes. We report that our model achieves 0.734 mloU and 0.072 RMSE while MulMON achieves 0.489 mloU and 0.069 RMSE. The experimental results show that our model is capable of handling challenging geometries with photorealistic observations. The metrics can be further improved by the per-object finetuning to 0.870 mloU and 0.031 RMSE. The comparison between ground truth observation (GT), S0OC3D inference results (SO0C3D) and the per-object finetuning results (S00C3D+) in Fig. p shows that the per-object finetuning process capture the fine detail of objects leading to more accurate instance masks. See Appendix for more qualitative results.
ring layout
grid layout
layout
Figure 7: Demonstration scene editing and ren- dering from our learned object representations. Note that all three images are rendered by the same set of learned objects representations.
color
shape
size
Figure 8: Latent variable traversal.
S00C3D+
SOOC3D
Figure Qualitative comparison before and after per-object finetuning.
LIMITATION
We observe that our model may fail to model objects located on the boundaries of camera visual cones. This challenge is rarely discussed in previous 3D aware object-centric learning literature since they either process all frames once (Kabra et al., [02T Henderson Lampert 2020) or the scenes are not large enough to trigger the problem (Li et al. 22200 Stelzzer et al. 2021] Yu et al. 2022). In cognitive map, objects are abstracted as points. A large object can be partially observed while its predicted centers are still outside the current view. As result, the object is not retrieved from the cognitive map. We anticipate object. Then the cognitive map should retrieve any object whose bounding box intersects with the visual cone for the current view. We leave the bounding box prediction as our future work.
this problem can be handled by predicting bounding box for each
6 CONCLUSION AND FUTURE WORK
We propose framework for unsupervised 3D object-centric learning for handling scenes of large scale and a varying number of objects the scene. We introduced factorized latent learning which separates the camera pose and view-invariant object latents. Our abject-compositional nerf allows the learning of 3D representation in the object camera coordinate system. The cognitive map ensures the framework keeps all the detected objects. Our learned view-invariant 3D object representation potentially be applied in the SLAM system or the robotic object manipulation of object representation. Our model cannot handle dynamic objects in the scene which is left our future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab, 2016. httppsss/aaxx...co////aa/a 03801
Christopher Burgess, Loic Matthey Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. ArXiv, abs/1901.11390, 01 2019. http::tts:///aiiiii
abs
B. Michael Chang, Tomer Ullman, Antonio Torralba, and B. Joshua Tenenbaum. A compositional object-based approach to learning physical dynamics. ICLR, 2017.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes, 2021.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. AAAI, 33:3412- 21222-240000 07 2019. doi:
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. AAAI, 34:3684 24:3684-3692, 04 2020. doi: N0.1609/aai.vv34000 v34104.5777.
Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for efficient reinforcement learning. /CML, ICML '08, pp. 240-247, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 978160558206 doi: 001144//390156.1333
URL
https://doii http:///i..rg//...// 11w.....111190011
139C187
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for learning symmetric and disentangled multi-object representations. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2970-2981, PMLR, 18-24 Jul 2021. URL
http:/ http:://pocceeinngg. .mrr aeress/v133/ 3 emami 2 la html
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020a. URL
https://openreviier net
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction Bottlenecks in Object- Centric Generative Models. ICMI. Workshop on Object-Oriented Learning. 2020b.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement. arXiv preprint arXiv:2 2021.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 0233-3241, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838811
M. Ali Eslami, Danilo Jimenez Rezende, Frederie Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360(6394) 1204 1210. 2018. ISSN 0036-8075. doi: 10.1126/sciemce..ard 6394/1204]
URL https:::tsptcccoee www.:/eww...iiieicce...... orq/ ozq/content/360)
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NeurIPS, NIPS' 17, pp. 6694-6704, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860966
10

---

Under review as a conference paper at ICLR 2023
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loie Matthey, Matthew M Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
Paul Henderson and Christoph H. Lampert. Unsupervised object-centric video generation and decomposition in 3D. In NeurlPS, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. pp. 1988-1997, 07 2017. doi: 10.1 10.1.../.2..
A. Juliani, V. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar, and D. Lange. Unity: A general platform for intelligent agents. ArXiv,
2020.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. SIMONe: View-invariant, temporally-abstrrr abstracted object representations via unsupervised video decomposition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, net http://eeeeeettttoeuruaaaaaa---e TMnt
2021. URL httpttttt//ppeeeeeeeeeee
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12549 2021.
Rob Kitchin. Cognitive maps: What are they and why study them? Journal of Environmental Psychology, 14:1-19, 03 1994. doi: 10. 00.101660072--04440088111111
Nanbo Li, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi- object scenes from multiple views. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5656- 5666. Curran Associates, Inc., 2020. URL/attps.//prrceeediin wwwrips....e...ueeeee... www..iiiuc..pppepea........... cc/paper/
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In ICLR, 2020. URL https:://openeeveeeee wwwp:nrwwwwwww..ee...eeeeeeee nwt/ffornnt
1d-rk103ySYDH
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
Joseph Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In ICMI,, 07 2018.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1) dee 2021. ISSN 0001-0782, doi: 10.1 145/3503200 URL https::/doo....... org/
10.1145/3503250
Cameron Smith, Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B. Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields, 2022. URL
hittps://arxin www.xiiv..gg/aaa a 2205. 03923
Karl Stelzner, Kristian Kersting, and Adam R. Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation, 2021.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeck Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. arXiv preprint arXiv:2202.05223, 2022.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li. Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning composiiooaoi neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 13779-13788, 2021.
11

---

Under review as a conference paper at ICLR 2023
Hong-Xinn Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In International Conference on Learning Representations, 2022. URL / net hett/Forunn2dddnwWSSSSSSA.L
Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. arXiv preprint arXiv:2203. 1283, 1/283, 2022.
12

---

Under review as a conference paper at ICLR 2023
A
ALGORITHMS
Algorithm I: S00C3D Inference
Input: a cognitive map M. begin
/* iterate over views, T can be unbounded during test time x/
for 1. = 1.,,..., do
Receive posed observation (Oe., 2z);
Xçcene Encode(o...;
2t gxpcens;
// Gaussian distribution
/K Su, (i) is the global index of the ith object in view v x/
[(A de, (x)))=====xxxx Querr(M,,)))))) u,, = (Ait A Agt for = 0, 1. Lo
(Athat, wheree,0 xrres0)
Executed in parallel for i = 1.... K do whatt 4xphatts 2t,
// Gaussian distribution
whered R,
4x.h Wxwhere
// Gaussian distribution
pread 2,,
"xpresd ;
// Bernoulli distribution
Compute Dishat KL (9xphati (xxphar))))))
Compute Dwhere Dz DKL wheree [xxmerco));
13
14
Compute DKL,A D K D end
(gxpron (1aggran.);
DEL E (Dihht + Dichere) x pres. + KL
2,
C = - log p(o,, 1 1kkk 19kt ,4
whatt wheredd prest scone) ) + DKL;
/+ a is the auxiliary input specifiee fied in Appendix
X N/. fol=5t
whaatt S ait
whate wheret presd
Oup (r, a);
19
end
Register (((A L+1 &r, k,)))......,MMM....
20
end
end
23
return Cognitive Map M storing the object-centric scene representation.
13

---

Under review as a conference paper at ICLR 2023
Algorithm 2: Cognitive Map Register
Input: a cognitive map M, latents with global index {(A,, ((1)) camera pose v. begin
for i = 1.,,.... K do Xwhhat As; Xathat, Aathat X
P4 pothere, J..kere Aihhere
where
mueeee, u);
(i) = -1 and (2 = l)) > 0.5 then
=
Ob ject newly detected in this view
(i) + len(M);
M.append(X:):
else
+ Objectt detected by previous views M[o A
end
end
end
14 return Cognitive Map M storing the object-centric scene representation.
Algorithm 3: Cognitive Map Query
Input: cognitive map M,, camera pose v, global prior Aprior. begin
initialize empty list T= []:
* Find all existing object in view
for = O, 2,,, (en((M) Xathat Xphat, Awhhat >;; ph neheree owkere Xwheeee
do
if pu where in the FOV of and expres (zpres (2pres 1) 0.5 then v);
u where &- ProjecctT www.iaaaaooooooooonaaaaiiieae L.append((a..))
end end
10
Rank latents with the probability of existence and take the top K latents TopK (L, (z 1))
x/
11
12
while len(1) < K do L.append (Aprioor -1))
14 end
15 end
return L containing K queried object latents.
14

---

Under review as a conference paper at ICLR 2023
B ELBO OBJECTIVE.
Object Centric NeRF Likelihood. In our formulation, each pixel in one image emits a single ray. be computed from pixel
Thus, with known intrinsic parameters of the camera, ray directions coordinates. Each ray is associated with two observables, i.e. the color and the depth of the hit point of the ray. Following common practice, we assume the likelihood of color and depth are independent given latents Z.,.
For single-component NeRF, the color returned by ray r(.) originated from wo pointing towards direction e, i.e. r(1) = To + et, in a neural radiance filed is defined by the rendering equation
C(r) J. T())(((())(((()).c. e)dt,
(6)
where a(r(l)) and e(r(1),) are NeRF density and color at point r() separately, T(I) exp (- o (r (t)) dt') is the transmittance term (Mildenhall et al. 2021). Under mild assumptions,
the probability density that the observed colors originate at the depth t via the ray r(-) is given by p(t, = (())t))((t) (Stelzner et al.) 2021). An unbiased estimator is given by
logpo(t, z.. = log oo (r())2.) Evwgt) low (r (L')2.)) / (t')],
where z(-) is proposal distribution with support [0, 4). 0 is the parameters of a NeRF decoder.
By replacing in Eq. 7 with depth di, we can compute the likelihood p(dgl-) of the depth dg of pixel j. The color likelihood of pixel j is given by p(c(l-) = NNc, a.), where ét is the predicted color and o. is a constant variance. Thus, the per-pixel log likelihood decomposed as log p(o (2)) log p(dj 2,) + log p(ct Z.). For each pixel, we only need to evaluate the NeRF at the depth d'. and a random point sampled from q-)) Points beyond a fixed clipping depth are deemed unreliable and are discarded.
Under the object-centric learning setup, the object-componee cnpeettcomppiitionalll NeRF is a composition of K + 1 NeRFs. The NeRF density (r(()) = Ek (x((()) is the sum of the density of each component. (x(r(t)) (r()),, e) is now a weighted sum of individual components. The color é(r(t))= Ek (((())
Following common practice, we assume the likelihood of rays is independent given object latent. Then the joint likelihood p(O,, !Z,,) in Eq. 5 is the product of the likelihood of all rays. In practice, for each view/image, instead of using all rays/pixels, for efficiency reasons, we only sample a set of rays/pixels to compute the likelihood. By plugging the likelihood function into the Eq. 5 we get the ELBO that SOOC3D is trained to maximize.
The KL term Cyl in Eq. 5 crucial to the emerging of view-invariant object-centric representations. For each refinement iteration I at time step t, we compute the KL term as
pres, 2
[DKL( (9y (1 + Dr (kkllgspe (Agggere (1gg: + Dr (9sgrer (1 (8)
Recall that, at the beginning of the inference process the cognitive map is empty and, instead, global priors will be retrieved from the cognitive map. The global prior of whatt and wheeee takes the form of zero-mean isotropic Gaussian. To encourage the rejection of empty component, the global prior of zpres is a Bernoulli distribution with p(zpres = 1) x O. We further encourage the rejection of empty components by weighing Dx P ( ) and D (1y (fywer /1 (imeere..)) with prres As
consequence, rejected components will not induce KL penalty.
Crucially. the KL term encourages object latents remain constant across views. One local optimum for models maintaining dynamic set of object representations is to discard all previously detected objects and re-discover all objects in the current view (Crawford Pineau| (020). Instead of learning mode achieves high reconstruction
neewinnvriiant representations, the discoverrreconsstuuttiiisss likelihood by only extracting view- ddeendependent representations. The KL term in our objective discour- ages such local optimum since re-discovering an object induces high KL divergence between the global prior and the posterior.
Another common concern is that when one object is temporally blocked by other objects, it is excluded from the likelihood computation directly and can be discarded without changing the observation likelihood. The KI. term between geree preserves the existence of such blocked objects.
15

---

Under review as a conference paper at ICLR 2023
The refinement network takes into account both object location zhheeee and appearance information 2 whhat By minimizing the KL term between each update, we discourage any abrupt changes to our latents. The KL term between zuterr enforces better handling of multiple objects with the same appearance. Thus, exchanging the location of two objects will lead to a large KI. loss. Note that in practice two rigid bodies cannot occupy the same space. If two objects share the same space and have the same appearance information, our model will learn to reject one of them by setting the corresponding zPres since rejecting one will not decrease the likelihood.
C TRAINING
In this section, describe training pipeline. For each scene in the training set, SOOC3D first takes a set of T views sequentially as input and detects and refines object representations. Then the of object representations refined through T views is used for query view reconstruction. The likelihood of Q query views is computed before we get the ELBO as specified in Eq
While the cognitive map is key factor scalability, it also makes end-to-end training unstable. As introduced in the main paper, any object with p(zPres = 1) < 0.5 will be rejected during cognitive map registration and objects that are outside the current view will not be retrieved. Objects can be constantly dropped during the registration and query process at the beginning of the training stage introducing a large variance.
To overcome this issue, we resort to curriculum learning and are-train our model on small scenes that can fit into camera FOV (detailed below). During registration, all K object latent distributions are kept regardless of their gures value. Similarly, the query function skips the in-view check and returns all registered object latents. Thus, for each scene, we will have the same set K object throughout.
We stop the pre-training at 20, 000 iterations when our model can stably predict object locations and zpres value. Then we train our model on scenes with the full cognitive map functionalityy
D DATASETS
30
23
00
-2
-50
-5
16
15
Scene setup example. From left to right is the large, medium and small scene. Objects are marked in black and cameras are marked with colored triangles. Cameras are uniformly randomly sampled from camera rings (color circles).
Figure 10:
To evaluate the scalability of our approach, we construct large-scale dataset with Unity3D (Juiiaii et al. 2020) termed Unity dataset. The dataset contains scenes of three different scales termed as small, medium and large scenes used for evaluating the scalability of our approach. For small and medium scenes generate 40, 000 scenes for training and 1, 000 scenes for testing. For large scenes, we only generate 1, 000 scenes for testing.
For each small scene, 2 ~ 5 objects randomly placed in square of 5 unit width. 10 cameras of resolution 128 x 64 are randomly sampled on a ring surrounding the area. A medium scene twice the size of small scene where 3 ~ 10 objects randomly placed in a unit by 10 unit region. In particular, we treat each medium scene as the overlapping of three square areas, each of which of unit width. For each area, spawn 10 cameras in ring layout. Note that the size of the medium
16

---

Under review as a conference paper at ICLR 2023
scene is now too large to fit into camera FOV. Different cameras in medium scene may capture different set objects. In a large scene, 12 ~ 24 objects randomly spawn a 10 unit by 15 unit area, which is 3 times larger than the medium scenes. In terms of camera setup, decompose the large area into 10 regions, each of which a square area of 5 unit width. 10 cameras are set similarly ahove. Thus for a large scene, have in total 100 views available. The camera ring layouts illustrated in Fig. 10
During training, for each small scene, we randomly sample 3 input views and query views, For each medium scene, in each area, we randomly sample 2 input views and query view. For testing, for small and medium scenes, we keep the same number of input views and use the rest as query views. For large scenes, we randomly sample 2 input views in each area. A large-scene inference visualization shown in Appendix H
The Blender dataset is built in Blender with raytracing renderer. The object and camera arrangements follow the small scene setup of the Unity dataset. Each scene in the Blender dataset contains 3 objects of non-trivial structures. The training set contains 10K small scenes, each of which contains 10 views. The training procedure is identical to the one described above. Similarly, during training, for each scene, we randomly sample 3 input views and 2 query views. The Blender dataset contains 10, 000 training scenes and 200 testing scenes.
Unlike the Unity dataset, for the Blender dataset, render each view in 512 x 256 resolution. During training, we downsample observations to 128 64. For per-object finetuning, we use the original resolution.
E NETWORK STRUCTURE AND HYPERPARAMETERS
Auxiliary input:
We group the auxiliary input into three different bundles.
bundle
quantity
note
RGB and depth observation mask indicate whether the depth is beyond clipping plane ray direction in object local coordinate system
depth mask
data bundle
Fk
object local coordinate of surface sample object latent distribution grad. of A w.r.t loss depth likelihood of the composed NeRF predicted surface sample density
A;
latent bundle
dLx, log log
predicted surface sample density of the hthh NeRF component grad. of w.r.t loss
dLa
log p(c)
color likelihood of the composed NeRF color likelihood of the k NeRF component
inference bundle
log Pk (c) log lcc)/
Srave-one-out color likelihood predicted color
čk dCa.
predicted color of 4th NeRF component grad. of č. w.r.t loss
Hyperparamter:
quantity
value
learning rate
le
Omar
the maximum NeRF density
5
Te
color prediction var.
0.2
L
the number of AVT iteration
3
R
the number of sampled ray
2048
observation resolution whott dimension
64 128
D
64
17

---

Under review as a conference paper at ICLR 2023
(1288)
Ic3 (128)
Shift and ocale 3
(64)
for (3286
Shift and scale 4
(64)
(E5)
1:10 (64)
1:11 (64)
Figure 1I: NeRF decoder structure. We specify the output dim for each fully connected layer. fc1-4 has no activation function. The rest fully connected layer uses leaky relu. is the harmonic representation of point coordinates. a and e are the predicted raw density and color. is the ray direction.
inference
Convi (5, 32, 2, 1]
PartialConvt (5, 32, 2,
latent bundic
H--1
J-1
Mox pool (2)
Mast pool 12)
LSTM (128)
Conv2 (5, 32, 2, 1)
PartialComv2 32,
fr3l [128]
pool
fr2 (256)
fo4 (135)
(5, 32,
PartialCondd 32, 2,
fcll [512]
Conv4 (5, 3,
Conv5 (5, 64, 2)
Conv6 (5, 64,
Refinement network structure. For convolution and partial convolution layer specify (kernel size, output channel, padding, stride). For LSTM we specify the dimension of hidden state. It produces the additive delta of X;
Figure 12:
18

---

Under review as a conference paper at ICLR 2023
F RENDERING EQUATION FOR DEPTH, MASK AND UNCERTAINTY
To render per-component RGB, we apply the render equation Eq. 9 on individual NeRF components get the color. The transparency rendered using equation Eq. I0 The returned transparency always fails in the interval [0, 1].
Cx(r) = 1000000 (()))((t))))a(((()eeee
(9)
As(r) = [ (r(t)on((t))dd..
(10)
Finally, we lay C+(r) on checkerboard background with transparency (r).). The Eq the contribution made by each component the rendering of the composed NeRF.
11 computes
Ke(r)= J (r)))x((t))dd.
(11)
The segmentation mask is obtained by arg max, (K)(r)) Eq. 12 gives the rendered depth of view.
D(r) 1.00 (())a(((()td..
(12)
For the uncertainty map, we interpret the normalized raw density detailed in Sec 3.2 as the probability that point in space is occupied by matters. The view uncertainty is computed as by Eq. [3 where H is the entropy function.
H(r) / T())tt(t)
(13)
19

---

Under review as a conference paper at ICLR 2023
G ADDITIONAL BASELINE.
In this section, we discuss ObSuR ObSuRFStelzner Stelzner et al. (022)) as an additional baseline. ObSuRF is a 3D scene object-centric learning model based on the slot attention Locatello al. (2020) mechanism. ObSuRF relies on a global coordinate system and by design only takes observations from one view as input. Thus, ObSuRF
directly comparable to our method.
Since slot attention is an iterative object-centric learning algorithm shown effective on 2D, 3D and video object-centric learning tasks (Stelzner et al. 2021] Locatello al.) 2020] Kipf al. 2021). We attempted to replace the amortized variational inference process in our model with slot attention. The slot attention model structures are identical the one used in ObSuRF and we decompose each slot into zithere, 2i zires whot for each object. We further allow slots to be carried over to the next frames (with projected 2 zwhere) for multi-view online updating. The training likelihood function remains unchanged and the KL term replaced with 1,2 distance between latents.
After training the slot attention-based model in small scenes, we match the ground truth object identities with the predicted object identities after each input view. We report that the slot attention module only preserves 27% object identities average per update, while our full model equipped with amortized variational inference preserves identities all the time. Further investigation reveals that during subsequent slot attention iteration, the identities, well representations of slots discarded and assigned.
The slot attention can be interpreted clustering algorithm operating feature vectors extracted from observations. We conjecture that the feature vectors are view-dependent and the clustering mechanism fails to convert view-dependent features into view-independent object representations. Thus, compared with a multi-view setup, slot attention is better suited for continuous frame processing where the correlation between consecutive frames large.
To compare with ObSuRF, we downgrade our model to the single view inference mode by setting the number of input views and the number of query views be one and train the CLEVR3D and the MultiShapeNet dataset Stelzner et al. (2021).
Table 2: Quantitative results on CLEVR3D and MultiShapeNet dataset. ObSuRF performance is obtained from the original paper.
CLEVR-3D
MultiShapeNet
MSE x Fg. Depth MSE Fg-ARI
ARI
MSE 10 Fg Depth MSE Fg-ARI
ARI
ObSuRF
0.78
0.10
95.7
94.6
1.91
3.44
81.4
64.1
Ours
0.84
0.12
96.2
92.2
2.04
3.81
83.1
61.4
Our model achieves performance par with ObSuRF. Note that model is not designed for the single view scenario.
20

---

Under review as a conference paper at ICLR 2023
H ADDITIONAL RESULTS ON UNITY DATASET
Below we show per-object NeRF finetuning results on the Unity dataset. We show the ground truth data (GT), SOOC3D inference results (S00C3D) as well the results of 5000 iteration per-object NeRF finetuning (S00C3D+). While S0OC3D can capture the structure of objects, the finetuning process recovers more details like edges or shadings. Most importantly. the object identities are preserved throughout the finetuning process.
Note that per-object NeRFs also built on the object local coordinate system. Thus, one can finetune an unboundedly large scene and register each per-object NeRF into the cognitive map to achieve unsupervised unboundedly scalable high-quality objecc-compositional NeRF reconstruction.
SOUC3D+
S00C3D
GT
S00C3D+
S00C3D
Figure 13: Qualitative comparison before and after per-object finetuning.
21

---

Under review as a conference paper at ICLR 2023
To demonstrate the scene scale scalability, below show long inference sequence (without per-object finetuning) in a large scene in Fig. 14 and Fig. 15
Cog. Map
Ground Truth
Reconstruction
Figure 14: Visualization of steps 1-10 20-step inference trajectory.
22

---

Under review as a conference paper at ICLR 2023
Cog. Map
Ground Truth
Reconstruction
Figure 15: Visualization of steps 11-20 a 20-step inference trajectory.
23

---

Under review as a conference paper at ICLR 2023
1 ADDITIONAL RESULTS ON BLENDER DATASET
Fig. sooss the qualitative comparison between our methods and MulMoN[Liet al. (2020) baseline. Both our model (S00C3D) and MulMON are trained with low-resolution observation (128 64). As NeRF-based method, our model can render images of arbitrary resolution, which allows us to finetune the per-object NeRF with higher resolution observation (512 256).
When facing non-trivial geometries of varying sizes, our model can capture object structures ac- curately with correct pose prediction. Notably, after per-object NeRF finetuning, fine details are recovered leading to more accurate RGB, depth as well instance mask prediction. While thin object parts like the legs of round tables are not reflected on the rendered instance masks (the main source of mloU error) due to NeRF sampling interval length, we'd d like to point out that they are correctly modeled by our SOOC3D+ as shown their RGB reconstructions.
MulMON
S00C3D+
SOOC3D
S00C3D+ Components
Qualitative comparison between the inference results of baseline (MulMON), the inference results of our model (S0OC3D) and our per-object finetuning results (S0OC3D+).
Figure 16:
24

---

Under review as a conference paper at ICLR 2023
Below show more SOOC3D+ results.
(a) RGB observation
(b) depth observation
(c) Instance mask
(d) S0OC3D+ RGB reconstruction
(e) S00C3D+ depth reconstruction
(f) S00C3D+ segmentation
Figure 17
RGB observation
(b) depth observation
(c) Instance mask
(d) SOOC3D+ RGB reconstruction
(e) S00C3D+ depth reconstruction
(I) S0OC3D+ segmentation
Figure 18
25

---

Under review as a conference paper at ICLR 2023
(a) RGB observation
(b) depth observation
(c) Instance mask
(d) S00C3D+ RGB reconstruction S00C3D+ depth reconstruction
(f) S00C3D+ segmentation
Figure 19
26


Reviews:
[{"review_id": "osHzyoCC0h7", "reviewer": "Reviewer_zLxX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a 3D object-centric representation learning on potentially unbounded scenes. The method infers object poses and view-invariant object representations in object coordinate system using RGBD data. Amortized variational inference is used to process sequential input and update object representations online, and formulated as an ELBO objective. The results are presented on synthetic datasets and the metrics show that the method outperforms MulMON, a prior art for 3D object centric learning. ", "summary_of_the_review": "The paper attempts to solve a important problem of incremental object learning in multi-object setting. However, the contribution to the learning method is incremental. given prior work. The probability densities should be connected to the underlying object and scene geometry and the formulation should be explained clearly. Finally, more ablation studies could be used to explain individual design choices.", "strength_and_weaknesses": "Incrementally learning object representations over time in a multi-object setting is an important and timely problem. This paper does a good job in formulating the problem as a latent probabilistic inference. The paper describes several non-trivial concepts quite lucidly. \n\nThe main issue I noticed was the similarity to the prior work MulMON. I read both papers carefully, and I find only incremental modifications in the paper. \nThe ideas presented as novel aspects in the paper are also not clearly motivated/explained. Particularly, the camera pose and the NeRF models are expressed through latents, but how are they connected to the underlying 3D geometry? The formulation appears to be high-level, and might not be relevant to real-world problems. It would be useful to make connections to practical techniques such as iMAP (ICCV 2021), which also look to solve the incremental scene mapping problem. \nThe paper also claims to be able to handle unbounded scenes. But there are assumptions that appear to violate that flexibility. For example, the method uses RGBD data, (instead of RGB that is typically used in NeRFs) and truncates depth values larger than the clipping plane. It also uses global camera pose, which implicitly means that the scene is bounded.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i945NFOZLrI", "reviewer": "Reviewer_bmFc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a scalable generative-model-based unsupervised 3D object-centric learning framework, \n   that can learn scenes of arbitrary numbers of objects from multi-view camera observations.\nA generative model is presented. Various experiments are performed on toy simulated dataset of a dozen basic 3D objects. \n\nUnfortunately claims are overstated, e.g. scalability, similar classical methods are unreferenced.\n A large number of assumptions regarding the toy scene scenarios needs to be described in the limitations section.", "summary_of_the_review": "The authors method is interesting, relevant and appears to work in the specific context, unsupervised learning.\n\nUnfortunately the authors claims of novelty and model capabilities are missing references to similar prior work and ideas and overstated.\nThe claims of novelty are:\n1) the first unbounded scalable generative-model-based unsupervised 3D object-centric learning framework. \n2) view-invariant object representation\n3) store a potentially unbounded number of objects detected for scalable inference\n\nThese are overstated considering the mathematical representation used and the assumptions of the method, as follows.\n\nIn my understanding, the model and mathematical data representation is as follows\nIn the 2D image plane: \n   each object is described by a 2D coordinates location and orientation\n     - images are assumed to be upright\nIn 3D: \n    - objects are assumed to be upright, limited in number (not unbounded), lying on a plane in 3D\n             thus described by 2D coordinates in a ground plane, and a single in-plane rotation.\n   - in a fixed rigid configuration (e.g. no motion)\n\nThis representation is essentially mathematically equivalent to previous work in unsupervised viewpoint-invariant learning has been achieved with the same mathematical description[a], contradicting claims 1) and 2)\n\nThe authors then claim “We assume there are at most K objects”, which contradicts claims 1) and 3) of unboundedness.\n\n* Limitations should be more accurately discussed. The authors state the only limitation is that objects out of view cannot be modeled.\n    \"We observe that our model may fail to model objects located on the boundaries of camera visual cones.\"\nThis is inaccurate, given the large number of limitations and assumptions, e.g. rigid scene, existence of a ground plane, upright camera, limited number of objects. The authors need to more accurately discuss the real limitations of the method.\n\n* Related: the authors make use of the multiview NERF decoding representation.\nA new method PERF, achieves similar accuracy to NERF with no training, via basic gradient methods.\n[b] Rasmuson, Sverker, Erik Sintorn, and Ulf Assarsson. \"PERF: Performant, Explicit Radiance Fields.\" arXiv preprint arXiv:2112.05598 (2021).  Frontiers in Computer Science 2022\n\n\n", "strength_and_weaknesses": "Strengths\n- The authors method is interesting and relevant, unsupervised learning of 3D objects in a scene is a challenging task.\n\nWeaknesses\n- The claims of novelty and unboundedness are overstated\n- Some links to unsupervised learning from similar data representations and object localization are missing\n- Experiments are limited to toy scenes, few objects\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aza6BlxQo0N", "reviewer": "Reviewer_a3V1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a method for unsupervised object-centric learning from multiple images. The method is based on a number of techniques: generative modelling with variational inference and neural radiance fields (NeRF) to obtain 3D scene representations and rendering. In contrast to existing compositional and amortized NeRF models, this work proposes a latent object representation, called Cognitive Map with registration and querying mechanisms that allow inferring and reconstructing an unbounded number of objects. Results show how the method capable of segmenting the objects for larger scenes (in terms of spatial extent and number of objects) than those seen at training time, while other methods in the literature fail to generalize in such a way.", "summary_of_the_review": "SOOC3D allows obtaining a highly interpretable and controllable 3D object-centric representations from images. In contrast to literature, it tackles scenes that have variable (and possibly unlimited) number of objects using its novel Cognitive Map. Instead, most object-centric methods assume a fixed number of maximum number of objects. These are key strengths of this work that to my knowledge no other methods have. However, it has some theoretical and experimental limitations: The method posed as a generative model doesn't seem to be a proper model of the scenes it tackles. It's also unclear how the distribution over the binary presence latents is learned, its probabilities are used in an arbitrary way, and do not handle partially-viewed objects.\nThe method also needs a pre-training stage to make it work which makes it less useful for the practitioner.  Finally, as the literature is shifting towards more complex and real images, it would be useful to test the limits of this model on such data regimes (for instance on datasets such as the MSN dataset mentioned above). Unless the authors address my questions regarding its soundness, I believe this paper needs to address these design issues before being accepted.\n\n---\n\n**Post-rebuttal update**\n\nI thank the authors for addressing my questions in detail and the changes to the paper they commit to doing. I think they authors have a novel and interesting paper, and propose a number of useful techniques to make their method scalable. I think the contributions have the potential to being useful to the community.\nHowever, it still needs substantial changes before being ready.\n\nDiscrete latent variables of object presence are introduced as part of the iterative variational inference model, but the issues that this entails is not fully addressed. The authors refer to using a relaxed bernoulli distribution (which is still not cited nor explained in the updated manuscript) but little to no details are given (i.e. no annealing of the relaxed Bernoulli termperature, etc., use of non-differentiable terms, etc.). Instead, the authors claim that the distribution learns accurate presence probabilities thanks to the learning signal from the KL and log-likelihood terms. These claim does not seem substantiated. The presence distribution seem to be mostly learned during a pre-training phase using a hand-crafted simpler version of the datasets.\nThis work's main contributions would therefore be stronger if either:\n- the authors fully address the challenges of discrete modelling of objects (cf learning signal) and show the method works without pre-training.\n- and/or the method can be shown to work with substantially more challenging datasets. I do not think that dataset appearance complexity is orthogonal to the method in this work, since such increased complexity (textures, clutter, occlusion) complicates learning disentangled object representations (what, where, presence).\n\nSOOC3D+ relies on fitting a NeRF scene post-generation, and uses depth supervision loss that is also during training. As far as I understand, results and renders shown for SOOC3D+ therefore leverage privileged information (depth) to fit the scenes, which is unfair when comparing to methods that do not use depths. \nSpecifically, SOOC3D has worse RMSE than MulMon without finetuning, contradicting the claim that it complete outperforms MulMon.\nRegular volumetric rendering likelihood should be used in this finetuning stage as we cannot generally assume ground-truth depths at test time.", "strength_and_weaknesses": "**Strengths**\n- Literature in unsupervised learning commonly focus on scenes that fit within the camera's field-of-view which is a strong limitation. This work tackles scenes in which the stream of input views do not have that property.\n- Latent representation is disentangled and interpretable to a large extent due to its $z_{where}, z_{what}, z_{pres}$ structure. This enables better control the scene useful for downstream tasks.\n- Cognitive map is a novel mechanism to scale the model to an unbounded number of objects.\n- Experiments clearly show the ability of the model to handle larger scenes at test time.\n\n**Weaknesses**\n1. While the authors refer to their method as a generative model, proposing a model with variable sets of latents has some technical requirements, otherwise it is not well defined. In this case, I do not see how the proposed method can sample scenes and images because there is no explicit distribution over e.g. the number of objects per scene.  Since the model cannot control the number of objects of a scene, this is instead dealt in an ad-hoc way through its inference mechanism. For instance by choosing $q(z_{pres} = 1) < 0.5$ to be the threshold of discarding presence.  \n2. Related to the point above, the authors should be clearly explain how the discrete distribution over $z_{pres}$ (Bernoulli) is learned. Due to its non-differentiable nature, what is the signal to learn the logits? Do they use continuous relaxation, REINFORCE, or some other approximation (or rather sum the latents out)?\n3. SOOC3D requires a first pre-training stage (curriculum learning) to work well in practice and should be referenced from the description of the method in the main text. \n4. A particular weakness to the method, acknowledged by the authors, is that partially viewed objects are not handled properly when querying the cognitive map. This is an issue that specially arises as a consequence of the choice of query mechanism by object position and it seems to be a potentially important limitation in many types of scenes. \n5. This work assumes RGBD data and requires precise camera information (due to use of NeRF), this may limit its applicability to only some datasets. An ablation of the method without using depths (as the model doesn't technically seem to require depths) would be useful.\n6. The refinement network takes decoded NeRF images as part of its input, this can result in expensive inference as rendering with NeRF can be notoriously slow. Can the authors comment on the computational efficiency of inference at test time? \n7. As this work attempts to tackle larger-scale scenes, it seems important to experiment on more realistic datasets that have (a) larger and cluttered number objects, and (b) more complex backgrounds (which is often a challenge with self-supervsied segmentation methods). An example of such is the MultiShapenet (MSN) dataset used in [1].\n\n**Further questions/suggestions**\n1. What type of rendering is used with NeRF at test time in the experiments? Does the NeRF fine-tuning process use the train-with-depth loss or is it based on volumetric rendering loss? (this question is for both training and test time)\n2. Have the authors experimented with an ablation without the Cognitive Map? Does it result in worse/better performance?\n3. While the authors suggest that the model performs on par with Obsurf on CLEVR3D and the\nMultiShapeNet, the numbers seems to show a consistently worse performance. Would the authors elaborate what might explain the difference in performance?\n4. The deterministic slot attention baseline is great but it would be great to provide further reconstruction results to establish the usefulness of the method's iterative variational inference. While they state identities are not well preserved, this may not be an issue if the segmentations at the final iteration are strong (FG-ARI), please provide those results if available.\n\n[1] Sajjadi et al. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KeJ6RKtzEY0", "reviewer": "Reviewer_1tCi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a framework for scalable 3D object-centric learning. Unlike existing works that are limited to the bounded scene, this work allows to model 3D objects present in the large-scale 3D scenes. Specifically, the method mains a cognitive Map that allows the registration and querying of objects on a global map. This enables modeling 3D models in large-scale scenes. For each 2D observation, the method learns to radiance method in an object-centric coordinate system. Furthermore, the method models the object individually. The paper shows promising results on a synthetic dataset. ", "summary_of_the_review": "The presented method has a good motivation -- aiming at scalable 3D object-centric learning. But however, the paper is of low clarity. And more importantly, the experimental results/analysis do look not very strong as I detailed above. I thus tend to reject this paper. But I would like to hear back from the author during the rebuttal just in case I misunderstand anything. ", "strength_and_weaknesses": "## Strength\n- Scalable object-centric learning is very interesting and novel to the community. \n- The experiments in the synthetic dataset are promising.\n\n## Weaknesses.\nThe paper has several weaknesses which I'll detail below:\n### Writing\nMaybe I missed something, I have difficulties following the method details. Specifically,\n- I'm not sure about the role of amortized variational inference. If the method models the object individually, why is this step needed?\n- In sec 3.1, the paper claims the presented method is a generative model. But this is confusing to me. Because I didn't see any description of latent space. Is this an auto-decoder/generative latent optimization since I didn't see any encoder?\n- The workflow is not very clear to me.\nI would recommend restructuring the method section. Particularly, I would recommend better motivation when proposing a new module. This might improve the clarity a lot. \n\n### Method \nThe paper claims the advantage of dealing with large-scale scenes compared with other works. However, this work has lots of limitations. It's limited to simple objects, unlike other works that model realistic datasets.  \n\n### Experiments\n- The dataset shown in the paper is too simple. This makes doubt the ability of the presented method. I would recommend having more photorealistic datasets like Kubric. It would be convincing. \n- Given that the proposed method is a generative model, I would be curious about the latent space. Interpolation analysis might be helpful here. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject