Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes Object Scene Representation Transformer, a framework for learning scene representation. The framework combines scene representation with the Slot Attention that constructs a bottleneck such that object-centric representation naturally emerges from learning. The proposed method is fast and achieves good performance.
strengths_and_weaknesses: Let me start by honestly stating that my research background is in representation learning not in 3D understanding or rendering, therefore my comments should not serve as the key evidence for deciding the outcome of this submission.

The paper is well written and easy to understand even for someone who doesn’t work in the field. The technical details are sound and the claims are fair. The experimental evaluation seems comprehensive. The performance looks good.

As my background is in representation learning, I’d like to raise a few questions on its application in real-world and complex scenes. Object centric representation is mostly learned through adding inductive biases into a system, such as the Slot Attention adopted in the paper. A bottleneck stage can force a neural network to group pixels into a compact representation, and object *naturally emerges*. However, in my opinion and from my research experience, nothing naturally emerges in a neural network – it does what it does because we, the creators, told it so. In other words, we achieve this grouping by injecting inductive biases into the architecture.

I have always been cautious about injecting inductive biases because doing so mostly helps *low data, low compute* regime, but many times harms a model’s capacity to scale to more data and more compute. Take the example of ViTs vs. CNNs – ViT models have much less inductive bias but they need more data to train and start outperforming CNNs (by a significant margin) as the model sizes grow. Inductive bias is a double-edged sword and any practice of adding inductive bias should be well motivated and examined with extreme caution. 

Back to the scope of this submission – I found that the datasets used here are rendered rather than from real-world. Naturally in these datasets object properties, such as numbers, lighting, materials, are randomized in a limited range. This setting seems to match the concerning low data, low compute criteria, therefore more inductive biases, especially dataset engineered ones, will unsurprisingly shine among baselines. It may not be the case for real-world data in which case the scalability and capacity of a model can truly be tested.

I’d like to note the above comments are made without a comprehensive investigation of common practice in similar papers and domains. I plan to take all reviewers’ comments into consideration, and after reviewers’ discussion a further recommendation will be made.

questions: See above.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work tackles the task of view-synthesis and extends a prior work “Scene Representation Transformer” by introducing a slot-based bottleneck representation. This allows the model to learn disentangled representations using only the standard view-synthesis objectives, and the learned disentanglement shown to be more accurate the prior unsupervised methods. Moreover, this representation also enables object removal (although not more general editing e.g. translation/scaling that prior NeRF based unsupervised disentanglement methods would allow).

The proposed approach is evaluated using the CLEVR and Multi-ShapeNet. (MSN) datasets where it outperforms a prior work ObSURF in both, view synthesis and scene decomposition accuracy. However, the bottleneck representation does limit the model expressivity and the learned model is less accurate at view-synthesis than the base SRT model.
strengths_and_weaknesses: Strengths:
+ The overall approach is simple and intuitive. The key idea of adding slots after the SRT encoder makes sense and it would be valuable for the community to see the results.

+ The ablations performed regarding the decoder architecture choice are informative. In particular, the proposed ‘slot mixer’ solution seems like a good compromise in retaining slot properties while being more efficient than the ‘broadcast’ approach in prior work.

+ The reported improvements ObSuRF are clearly significant and highlight the benefits of adding slots in the SRT network i.e. ability to handle variable views and more accurate results.

+ The ablation highlighting the role of view synthesis for learning accurate decomposition was very interesting, and shows necessity of using novel views for learning as opposed to mere auto-encoding objectives.


Weaknesses:
- While the ‘slot mixer’ vs ‘broadcast’ decoder is a useful contribution, the overall approach rather straight-forwardly combines ideas from prior works like UORF and ObSuRF with the SRT approach, and I feel the technical contribution here is a bit limited. That said, I still think this maybe a valuable (if obvious) combination for the community to see.

- As a downside of using the proposed slot representation, the quality of the synthesized views does suffer. While this is only highlighted in Table 2 on one dataset, it would be helpful. to see similar results across all datasets.

- (minor point) I feel the claim of “3000x faster” rendering is a bit misleading as this mostly comes from the speedup obtained by SRT over volume-rendering based methods, and isn’t really a contribution of this work (the ‘slot mixer’ vs ‘broadcast’ speedup could be attributed to this work, but that is a much smaller factor).
——

Overall, I think this is simple but obvious paper. On the one hand, I am not sure this work makes any remarkable technical contributions or empirical findings. But on the other hand, it represents a well executed and sensible combination which the community would benefit from seeing, and I would therefore lean towards accepting it.


questions: N/A
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The paper addresses the problem of unsupervised decomposition of scenes into a discrete set of objects. The idea is that novel view synthesis requires reasoning about the geometry which should also be helpful for discovering objects without additional supervision. A novel transformer-based approach for object centric 3D representation learning is introduced which aims to overcome limitations of previous methods regarding generalization to complex scenes due to high computational costs.
- The method uses a slot-attention mechanism in combination with a light-field rendering formulation using a transformer conditioned on a viewpoint-dependent latent representation derived from these slots. To avoid high computational costs, an early fusion scheme is presented that mixes slots early and thus has to run the decoder only once and not per slot.
- Compared to [27], the proposed approach does not require depth supervision for efficiency as it relies on an SRT [23] based decoder for efficient rendering instead of volumetric rendering approaches.
strengths_and_weaknesses: - Strengths
    - The introduction provides a good motivation towards the goal of learning about compositions and the geometry of scenes solely from observations. The related works section provides good context regarding neural rendering, object centric learning and specific differences to closest related works on object centric 3d learning.
    - Moving unsupervised 3d scene decomposition towards more complex and realistic scenes is an important goal with many applications. The proposed method demonstrates good performance on the more complex MSN-Hard dataset and could serve as a valuable baseline, especially in combination with the dataset's ground-truth instance labels.
    - The proposed approach is simple and reasonable. Its evaluation supports the stated claims regarding the presented model.
    - The experimental comparison to [27] (as well as the description of the attempts to train [5] in the supplementary) convincingly demonstrate the advantages of the proposed model in terms of accuracy and runtime, especially when going to more complex datasets.
    - The design of the ablations allows for a good analysis of the different components involved in the approach. The decoder analysis clearly demonstrates how the SM decoder improves decomposition over an SRT decoder [23] and runtime over an SB decoder. The experiments on the role of novel view synthesis for decomposition validate the hypothesis that novel view synthesis indeed helps object decomposition. Evaluating whether the approach is mainly segmenting by color cues is a good test for generalization capabilities of the approach. Scene editing experiments give a nice qualitative indication that slots indeed represent meaningful objects.
- Weaknesses
    - The novelty of the proposed method is limited, as it is a rather straightforward adaptation of slot attention [17] to SRT [23] with an early fusion scheme for faster decoding.
    - Qualitatively (Fig. 3 of the supplementary), the SB decoder still looks better than the proposed SM decoder, although this comes at a large computational cost. The quantitative results suggest that the SM decoder performs better, which makes me question the metric a bit.
questions: - Fig. 3 in the supplementary seems to indicate that the Spatial Broadcast (SB) decoder suffers less from oversegmenting. Qualitatively it also seems to be more accurate around object boundaries but the quantitative results in Tab. 3 say the opposite. What could explain this discrepancy and could the proposed Slot Mixer (SM) decoder benefit from an explicit background modeling to avoid these problems?
limitations: Limitations and potential negative societal impact have been addressed adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a method to infer an object-centric scene representation aware of 3D geometry from multiple views of a scene. Similar to other works, it uses slot attention to segment scenes into slots, and it uses a ray-casting approach to rendering 2D views of the represented scene. However, the authors propose a novel decoder, based on Scene Representation Transformers (SRT), that only requires a single evaluation per ray query regardless of the number of objects in the representation. This makes the model faster than previous alternatives, while also achieving good quantitative metrics on different datasets.
strengths_and_weaknesses: **Strengths:**

+ Clear presentation:

The problem, method and results are presented clearly and it is easy to follow the paper.

+ Fast approach to decoding 3D object-centric representations:

The model has much lower computational costs per iteration given that the number of evaluations per ray do not scale linearly with the number of objects. This makes the model fast and more scalable than other alternatives.
 
+ Good performance compared to ObjSURF:

The model outperforms ObjSURF both quantitatively and qualitatively on different datasets.


**Weaknesses:**

- Missing references:

The paper is missing numerous relevant references. Specifically, it is missing the whole body of work of Sungjin Ahn's team, with ROOTS[1] being directly related to this paper, and relevant work by others [2,3].

- Evaluation could be improved:

While the paper compares in three datasets of increasing difficulty, the comparison is mostly to ObjSURF. The paper only compares to UORF on the harder dataset, in which it does not work well due to the high number of objects and computational costs associated. While this is a good comparison to highlight the better scaling capabilities of the proposed model (and this is an important point), it does not resolve the issue of how this model compares to competing approaches when they can actually compete. An easy way to solve this would be to train UORF and other previous methods on the simpler datasets, or compare on one of the proposed datasets in the UORF paper. In fact, there are experiments with different backgrounds and more complex objects in that paper, that are similar to the proposed datasets in this paper.
It would also be interesting to understand the quality of the segmentations by reporting regular ARI metrics, instead of just FG-ARI metrics, as these can be misleading when objects are not segmented tightly.


**Detailed comments:**

Overall this is a promising submission. While the idea of having an object-centric SRT model is relatively straightforward, the proposed decoder is novel and has clear advantages in terms of computational costs while still being interpretable. However, the paper could be improved with a more thorough literature review and a better comparison to previous approaches based on this literature review. There are a few questions to be answered, but overall I believe this can be a good submission with some modifications. I am currently arguing for its acceptance, and I would be happy to increase my score if the authors address my comments.

**References:**
[1] Chen, Chang, Fei Deng, and Sungjin Ahn. "ROOTS: Object-Centric Representation and Rendering of 3D Scenes." J. Mach. Learn. Res. 22 (2021): 259-1.
[2] Castrejon, Lluis, Nicolas Ballas, and Aaron Courville. "INFERNO: Inferring Object-Centric 3D Scene Representations without Supervision." (2021).
[3] Anciukevicius, Titas, Christoph H. Lampert, and Paul Henderson. "Object-centric image generation with factored depths, locations, and appearances." arXiv preprint arXiv:2004.00642 (2020).

questions: - How is the background handled?
- Differently from Slot Attention, it seems that the initial slot representation is learned. How does that affect the model performance and how does it affect its generalization capabilities to an a different number of objects from training/testing?
- How does the model compare to UORF quantitatively on the simpler datasets proposed in that paper?
- The segmentations compared on the FG-ARI metric do not take into account that some object segmentations might be covering the background. How do the metrics change when using the ARI metric
limitations: The authors mention limitations of their work and ethical considerations. The limitations of their work could be better analyzed by conducting some of the experiments suggested in the Questions section.
review_title: null

Meta-review: metareview: The paper received positive leaning reviews (2x borderline accept, 1x weak accept, 1x accept). The meta-reviewer agrees with the reviewers' assessment of the paper.
confidence: Certain
Decision: Accept