Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a 3D object-centric representation learning on potentially unbounded scenes. The method infers object poses and view-invariant object representations in object coordinate system using RGBD data. Amortized variational inference is used to process sequential input and update object representations online, and formulated as an ELBO objective. The results are presented on synthetic datasets and the metrics show that the method outperforms MulMON, a prior art for 3D object centric learning. 
strength_and_weaknesses: Incrementally learning object representations over time in a multi-object setting is an important and timely problem. This paper does a good job in formulating the problem as a latent probabilistic inference. The paper describes several non-trivial concepts quite lucidly. 

The main issue I noticed was the similarity to the prior work MulMON. I read both papers carefully, and I find only incremental modifications in the paper. 
The ideas presented as novel aspects in the paper are also not clearly motivated/explained. Particularly, the camera pose and the NeRF models are expressed through latents, but how are they connected to the underlying 3D geometry? The formulation appears to be high-level, and might not be relevant to real-world problems. It would be useful to make connections to practical techniques such as iMAP (ICCV 2021), which also look to solve the incremental scene mapping problem. 
The paper also claims to be able to handle unbounded scenes. But there are assumptions that appear to violate that flexibility. For example, the method uses RGBD data, (instead of RGB that is typically used in NeRFs) and truncates depth values larger than the clipping plane. It also uses global camera pose, which implicitly means that the scene is bounded.


clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written and the formulation appears to be correct.
The novelty is questionable given the similarity to the prior work MulMON.
The lack of details related to 3D geometry would make it hard to reproduce the results of this paper.
summary_of_the_review: The paper attempts to solve a important problem of incremental object learning in multi-object setting. However, the contribution to the learning method is incremental. given prior work. The probability densities should be connected to the underlying object and scene geometry and the formulation should be explained clearly. Finally, more ablation studies could be used to explain individual design choices.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a framework for scalable 3D object-centric learning. Unlike existing works that are limited to the bounded scene, this work allows to model 3D objects present in the large-scale 3D scenes. Specifically, the method mains a cognitive Map that allows the registration and querying of objects on a global map. This enables modeling 3D models in large-scale scenes. For each 2D observation, the method learns to radiance method in an object-centric coordinate system. Furthermore, the method models the object individually. The paper shows promising results on a synthetic dataset. 
strength_and_weaknesses: ## Strength
- Scalable object-centric learning is very interesting and novel to the community. 
- The experiments in the synthetic dataset are promising.

## Weaknesses.
The paper has several weaknesses which I'll detail below:
### Writing
Maybe I missed something, I have difficulties following the method details. Specifically,
- I'm not sure about the role of amortized variational inference. If the method models the object individually, why is this step needed?
- In sec 3.1, the paper claims the presented method is a generative model. But this is confusing to me. Because I didn't see any description of latent space. Is this an auto-decoder/generative latent optimization since I didn't see any encoder?
- The workflow is not very clear to me.
I would recommend restructuring the method section. Particularly, I would recommend better motivation when proposing a new module. This might improve the clarity a lot. 

### Method 
The paper claims the advantage of dealing with large-scale scenes compared with other works. However, this work has lots of limitations. It's limited to simple objects, unlike other works that model realistic datasets.  

### Experiments
- The dataset shown in the paper is too simple. This makes doubt the ability of the presented method. I would recommend having more photorealistic datasets like Kubric. It would be convincing. 
- Given that the proposed method is a generative model, I would be curious about the latent space. Interpolation analysis might be helpful here. 


clarity,_quality,_novelty_and_reproducibility: From my viewpoint, the paper is of high novelty. But it lacks clarity and thus decreases reproducibility. 
summary_of_the_review: The presented method has a good motivation -- aiming at scalable 3D object-centric learning. But however, the paper is of low clarity. And more importantly, the experimental results/analysis do look not very strong as I detailed above. I thus tend to reject this paper. But I would like to hear back from the author during the rebuttal just in case I misunderstand anything.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents a method for unsupervised object-centric learning from multiple images. The method is based on a number of techniques: generative modelling with variational inference and neural radiance fields (NeRF) to obtain 3D scene representations and rendering. In contrast to existing compositional and amortized NeRF models, this work proposes a latent object representation, called Cognitive Map with registration and querying mechanisms that allow inferring and reconstructing an unbounded number of objects. Results show how the method capable of segmenting the objects for larger scenes (in terms of spatial extent and number of objects) than those seen at training time, while other methods in the literature fail to generalize in such a way.
strength_and_weaknesses: **Strengths**
- Literature in unsupervised learning commonly focus on scenes that fit within the camera's field-of-view which is a strong limitation. This work tackles scenes in which the stream of input views do not have that property.
- Latent representation is disentangled and interpretable to a large extent due to its $z_{where}, z_{what}, z_{pres}$ structure. This enables better control the scene useful for downstream tasks.
- Cognitive map is a novel mechanism to scale the model to an unbounded number of objects.
- Experiments clearly show the ability of the model to handle larger scenes at test time.

**Weaknesses**
1. While the authors refer to their method as a generative model, proposing a model with variable sets of latents has some technical requirements, otherwise it is not well defined. In this case, I do not see how the proposed method can sample scenes and images because there is no explicit distribution over e.g. the number of objects per scene.  Since the model cannot control the number of objects of a scene, this is instead dealt in an ad-hoc way through its inference mechanism. For instance by choosing $q(z_{pres} = 1) < 0.5$ to be the threshold of discarding presence.  
2. Related to the point above, the authors should be clearly explain how the discrete distribution over $z_{pres}$ (Bernoulli) is learned. Due to its non-differentiable nature, what is the signal to learn the logits? Do they use continuous relaxation, REINFORCE, or some other approximation (or rather sum the latents out)?
3. SOOC3D requires a first pre-training stage (curriculum learning) to work well in practice and should be referenced from the description of the method in the main text. 
4. A particular weakness to the method, acknowledged by the authors, is that partially viewed objects are not handled properly when querying the cognitive map. This is an issue that specially arises as a consequence of the choice of query mechanism by object position and it seems to be a potentially important limitation in many types of scenes. 
5. This work assumes RGBD data and requires precise camera information (due to use of NeRF), this may limit its applicability to only some datasets. An ablation of the method without using depths (as the model doesn't technically seem to require depths) would be useful.
6. The refinement network takes decoded NeRF images as part of its input, this can result in expensive inference as rendering with NeRF can be notoriously slow. Can the authors comment on the computational efficiency of inference at test time? 
7. As this work attempts to tackle larger-scale scenes, it seems important to experiment on more realistic datasets that have (a) larger and cluttered number objects, and (b) more complex backgrounds (which is often a challenge with self-supervsied segmentation methods). An example of such is the MultiShapenet (MSN) dataset used in [1].

**Further questions/suggestions**
1. What type of rendering is used with NeRF at test time in the experiments? Does the NeRF fine-tuning process use the train-with-depth loss or is it based on volumetric rendering loss? (this question is for both training and test time)
2. Have the authors experimented with an ablation without the Cognitive Map? Does it result in worse/better performance?
3. While the authors suggest that the model performs on par with Obsurf on CLEVR3D and the
MultiShapeNet, the numbers seems to show a consistently worse performance. Would the authors elaborate what might explain the difference in performance?
4. The deterministic slot attention baseline is great but it would be great to provide further reconstruction results to establish the usefulness of the method's iterative variational inference. While they state identities are not well preserved, this may not be an issue if the segmentations at the final iteration are strong (FG-ARI), please provide those results if available.

[1] Sajjadi et al. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR 2022

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written and clear. The method builds on solid existing frameworks and methods (i.e. VAEs, Obsurf for its compositional nature and training-with-depth procedure and NeRF). I think this combination can be important to obtain practical and useful models. The framing of the method as a generative model, while having varying number of latents that aren't handled by the model, limit its technical soundness. It would be useful in order to understand the model to provide a diagram of the generative model and inference as is typically done in the literature.
The proposal of using the Cognitive Map is novel and the experimental results are significant. The pseudo algorithms provided and training details make this work reproducible.
summary_of_the_review: SOOC3D allows obtaining a highly interpretable and controllable 3D object-centric representations from images. In contrast to literature, it tackles scenes that have variable (and possibly unlimited) number of objects using its novel Cognitive Map. Instead, most object-centric methods assume a fixed number of maximum number of objects. These are key strengths of this work that to my knowledge no other methods have. However, it has some theoretical and experimental limitations: The method posed as a generative model doesn't seem to be a proper model of the scenes it tackles. It's also unclear how the distribution over the binary presence latents is learned, its probabilities are used in an arbitrary way, and do not handle partially-viewed objects.
The method also needs a pre-training stage to make it work which makes it less useful for the practitioner.  Finally, as the literature is shifting towards more complex and real images, it would be useful to test the limits of this model on such data regimes (for instance on datasets such as the MSN dataset mentioned above). Unless the authors address my questions regarding its soundness, I believe this paper needs to address these design issues before being accepted.

---

**Post-rebuttal update**

I thank the authors for addressing my questions in detail and the changes to the paper they commit to doing. I think they authors have a novel and interesting paper, and propose a number of useful techniques to make their method scalable. I think the contributions have the potential to being useful to the community.
However, it still needs substantial changes before being ready.

Discrete latent variables of object presence are introduced as part of the iterative variational inference model, but the issues that this entails is not fully addressed. The authors refer to using a relaxed bernoulli distribution (which is still not cited nor explained in the updated manuscript) but little to no details are given (i.e. no annealing of the relaxed Bernoulli termperature, etc., use of non-differentiable terms, etc.). Instead, the authors claim that the distribution learns accurate presence probabilities thanks to the learning signal from the KL and log-likelihood terms. These claim does not seem substantiated. The presence distribution seem to be mostly learned during a pre-training phase using a hand-crafted simpler version of the datasets.
This work's main contributions would therefore be stronger if either:
- the authors fully address the challenges of discrete modelling of objects (cf learning signal) and show the method works without pre-training.
- and/or the method can be shown to work with substantially more challenging datasets. I do not think that dataset appearance complexity is orthogonal to the method in this work, since such increased complexity (textures, clutter, occlusion) complicates learning disentangled object representations (what, where, presence).

SOOC3D+ relies on fitting a NeRF scene post-generation, and uses depth supervision loss that is also during training. As far as I understand, results and renders shown for SOOC3D+ therefore leverage privileged information (depth) to fit the scenes, which is unfair when comparing to methods that do not use depths. 
Specifically, SOOC3D has worse RMSE than MulMon without finetuning, contradicting the claim that it complete outperforms MulMon.
Regular volumetric rendering likelihood should be used in this finetuning stage as we cannot generally assume ground-truth depths at test time.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a scalable generative-model-based unsupervised 3D object-centric learning framework, 
   that can learn scenes of arbitrary numbers of objects from multi-view camera observations.
A generative model is presented. Various experiments are performed on toy simulated dataset of a dozen basic 3D objects. 

Unfortunately claims are overstated, e.g. scalability, similar classical methods are unreferenced.
 A large number of assumptions regarding the toy scene scenarios needs to be described in the limitations section.
strength_and_weaknesses: Strengths
- The authors method is interesting and relevant, unsupervised learning of 3D objects in a scene is a challenging task.

Weaknesses
- The claims of novelty and unboundedness are overstated
- Some links to unsupervised learning from similar data representations and object localization are missing
- Experiments are limited to toy scenes, few objects

clarity,_quality,_novelty_and_reproducibility: Some links to unsupervised learning from similar data representations and object localization are missing.

Unsupervised Generative Model-based 3D Object-centric Learning from 2D images:
[a] Toews, M., & Arbel, T. (2007, October). Detecting and localizing 3D object classes using viewpoint invariant reference frames. In 2007 IEEE 11th International Conference on Computer Vision (pp. 1-8). IEEE.
Specifically:
 - the viewpoint invariance mechanism is identical (i.e. a viewpoint-invariant reference frame defined in 2D and 3D by a location and orientation angle) to [a]
- the method of localizing object centers via Gaussian weighting is key to this method (Top of Page 5) log wg(xk) − SG(log wg(xk)),
      and is essentially similar to a difference-of-Gaussian used in [a]
 
summary_of_the_review: The authors method is interesting, relevant and appears to work in the specific context, unsupervised learning.

Unfortunately the authors claims of novelty and model capabilities are missing references to similar prior work and ideas and overstated.
The claims of novelty are:
1) the first unbounded scalable generative-model-based unsupervised 3D object-centric learning framework. 
2) view-invariant object representation
3) store a potentially unbounded number of objects detected for scalable inference

These are overstated considering the mathematical representation used and the assumptions of the method, as follows.

In my understanding, the model and mathematical data representation is as follows
In the 2D image plane: 
   each object is described by a 2D coordinates location and orientation
     - images are assumed to be upright
In 3D: 
    - objects are assumed to be upright, limited in number (not unbounded), lying on a plane in 3D
             thus described by 2D coordinates in a ground plane, and a single in-plane rotation.
   - in a fixed rigid configuration (e.g. no motion)

This representation is essentially mathematically equivalent to previous work in unsupervised viewpoint-invariant learning has been achieved with the same mathematical description[a], contradicting claims 1) and 2)

The authors then claim “We assume there are at most K objects”, which contradicts claims 1) and 3) of unboundedness.

* Limitations should be more accurately discussed. The authors state the only limitation is that objects out of view cannot be modeled.
    "We observe that our model may fail to model objects located on the boundaries of camera visual cones."
This is inaccurate, given the large number of limitations and assumptions, e.g. rigid scene, existence of a ground plane, upright camera, limited number of objects. The authors need to more accurately discuss the real limitations of the method.

* Related: the authors make use of the multiview NERF decoding representation.
A new method PERF, achieves similar accuracy to NERF with no training, via basic gradient methods.
[b] Rasmuson, Sverker, Erik Sintorn, and Ulf Assarsson. "PERF: Performant, Explicit Radiance Fields." arXiv preprint arXiv:2112.05598 (2021).  Frontiers in Computer Science 2022
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Object-centric learning is typically done in a 2d/latent space to learn disentangled representations from sensory observations. This paper studies the problem within a 3D context with a major relaxation compared to previous work -- that there could be an unbounded number of objects in the scene during test time (but assumes upto K objects in current view). This is the problem of 'object permanance', a long standing open problem in computer vision and cog sci/neuroscience. There is currently no known solution to this problem, even in toy domains. So at a high level, I think think this problem is quite interesting for the community to tackle with a lot of real world implications.

This model proposes an approximate generative model using variational inference and neural radiance fields. In order to handle object permanance over a potentially unbounded set of objects, authors propose a 'Cognitive map' inspired architecture to register and track representations over viewpoints. The end result is that the method is able to segment multiple objects in a scene, with the corresponding radiance field for each object. All experimental results are show on toy rendered scenes. 

There is consensus that the proposed approach studies a novel setting unlike prior approaches and shows reasonable experimental validation to provide its effectiveness to some extent. 

However, many questions remain unanswered before the paper is ready for publication. For instance there were questions about the generative nature of this approach, rigorous foundations for why certain choices were made and clarity of writing. While it is informative and necessary to study a toy domain, I would encourage the authors to perform experiments in some real image dataset as suggested by the reviewers. Since the approach is pitched to work with unbounded scenes, I think it is necessary to show some scaling on real world domains, even if they were directional. Otherwise it will be difficult for anyone, including the authors, to have conviction in all design choices and also make it difficult for other researchers to build on this paper. 

Overall I believe the paper studies an extremely important and interesting problem but it is not yet ready for publication.
justification_for_why_not_higher_score: Needs major revisions in terms of clarity, rigorous formulation of all design choices and more realistic domains for experimental validation.
Decision: Reject