Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel unsupervised scene decomposition model that infers object shapes, appearances and 3D poses. The benefits over existing models are the structured, 3D object representations which allows to manipulate objects in the scenes such as moving and replacing objects. This paper also shows that the inferred object representations can be used in a visual reasoning task.
main_review: Strengths:

+ The paper is clearly written and easy to follow.
+ It is novel and interesting to see object-centric learning models to infer 3D poses.
+ Validating the usefulness in downstream visual reasoning task is a plus.

Weakness:

My main conern is in the insufficient experiments that do not support the main contributions, i.e., the experiments do not demonstrate the benefits of the proposed 3D, structured object representations.

As for representations being structured (i.e., disentangled shape, appearance and poses):
- For disentanglement of shape and appearance, there is no experiments demonstrating this point. For example, this can be easily shown by swapping appearances of two objects in the scene while keeping their shapes unchanged, and vice versa.
- For disentanglement of 3D pose, the experiments include scaling and translation. However, there is no comparison to 2D baseline methods, which are able to do scaling and translation (at least the experiments on moving to the "right").

As for representations being 3D, there is no comparisons to show the benefits of 3D representation over 2D. In segmentation experiments, the results are worse than 2D baselines. In the visual reasoning task, the performance is worse than Aloe. Perhaps Aloe is better because it is designed for that task, but there is neither comparisons to 2D general object-centric baselines (such as original slot attention) or unstructured baselines.

Other minor weaknesses include:
- No ablation study on the model components to show the benefits of proposed technical contributions, such as the effect of regressing pose vs. not regressing pose (I suppose this should affect background decomposition because that is the only inductive bias that separates background slot from foreground slots).
- It would be good to demonstrate on datasets other than CLEVR.
summary_of_the_review: Although this paper is interesting in the ability to infer 3D properties of objects, the fundamental flaws in experiments are the main reasons that I recommend revision and resubmission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to decompose a scene into objects and infer the representations of 3D occupancy, color, and pose for each object from a single image of the scene without supervision. To this end, the paper proposes an autoencoding solution by combining the Slot Attention encoder with the GIRAFFE decoder. Each object is represented as a Neural Radiance Field (NeRF) additionally parameterized by the latent variables inferred from the encoder. The decoder then compositionally renders the objects. The experiments show that the proposed model (1) achieves competitive 2D segmentation performance on CLEVR6, (2) supports object-wise scene manipulation, and (3) outperforms non-object-centric methods on CATER snitch localization when combined with a powerful transformer.
main_review: The paper tackles a very challenging problem, is well-written and easy to follow, and has a good set of experiments for testing object-centric representations. However, my main concern is that the experiments did not focus on testing the '3D-ness' of the inferred representations.
- For scene segmentation, reconstruction, and manipulation, each scene is tested only on a single camera viewpoint. It is unclear whether the learned representations can capture the object appearance from viewpoints different from the input view. Previous work [1, 2] demonstrates this through novel view synthesis.
- The results on the CATER snitch localization task did not strongly support that the '3D' representations learned by the proposed model are more beneficial than 2D representations. One reason might be that this task relies more on long-term reasoning than on 3D reasoning. The authors may want to consider some other tasks that can clearly show the benefit of having 3D representations, like the ones used in [2]. Also, instead of comparing to Aloe, a more direct comparison would be to compare with Slot Attention combined with the same 12-layer transformer used for the proposed model.

**MINOR COMMENTS**
- The rendering function is denoted $g_\gamma$ in Equation 1 but $g_\tau$ in Section 2.1.
- From the paragraph directly below Figure 1, it seems the model can infer the camera location $c$. However, this was not demonstrated in any of the experiments.
- The object NeRFs are shared, which may only be able to deal with similar objects. The paper could be strengthened by experiments on more complex scenes, like the ones used in [1].
- What is the reference frame for the pose vectors? It is mentioned in Appendix B.1 that the manipulations are done by moving the objects forward on the Z or X axes. How are the XYZ axes defined? I would imagine that under different camera viewpoints, the same manipulation would have different effects, but the results in Figure 2 seem quite consistent.

[1] [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://arxiv.org/abs/2104.01148).

[2] [ROOTS: Object-Centric Representation and Rendering of 3D Scenes](https://arxiv.org/abs/2006.06130).
summary_of_the_review: I recommend reject, because the main claim, learning 3D representations, is not well supported by the experiments, and the benefit of the learned representations over prior work is not well demonstrated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model which is able to segment 3D scenes into objects by a combination of slot-attention (For inference) and a mixture of object NeRF functions which mix together (in 3D) to compose a scene. The method receives a single input image (with the camera coordinates though these are fixed) and extracts a set of slots - one slot for each object. These slots are decoded using a NeRF renderer: one part (the shape) generates the density, one (the appearance) generates the colours and one (the pose) transforms the points of the object to the appropriate pose in scene space. Results are demonstrated on CLEVR data as well as CATER (which is visually very similar) and some downstream tasks.
main_review: Strengths:

* an interesting combination of slot attention and NeRF mixture models.
* paper is nicely presented
* results are nice, though see below

Weaknesses:

While I like the paper in general I feel there are several points missing which are probably worth addressing to:

* The objects are transformed using the pose - but that means the appearance is transformed as well - this could be a problem if some of the appearance is pose dependent, for example shading will change, or reflections. Can the authors comment how this is handled here? my guess is that the post rendering network (the upscaling one) takes some of this burden but that's probably not ideal in terms of representation.

* How does the model perform when scenes are rendered from other viewpoints? if the resulting representation is indeed 3D it should be no problem to render the scene from other viewpoints - is that the case here? if not, why? (again, my guess is the upscaling network may be problematic here as well)

* I see that the masks used for estimating segmentation performance (Figure 3) are from the slot attention mechanism - what happens if you use the masks as decoded from the slots (i.e. render the densitiy of each slot separately). This would be more convincing to demonstrate these are "3D" segmentation.

* I feel the experimental validation is quite limited - using only a single dataset (even if it's two variants) is not very convincing as to the generality of the method. How does it work on other datasets? more/less complex.

* I would be happy to see more discussion of the post rendering / upscaling network - this seems like a cruicial part and is not discussed enough.
summary_of_the_review: All in all a nice paper but I think it suffers from some drawback as discussed above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model to infer structured 3D object representations from a 2D scene in an unsupervised fashion so as to represent the visual scene in an object-centric way. Specifically, the inference part adopts a similar mechanism as Slot Attention to derive object
slot latent code, and then maps slot latent code to 3D object representations with MLP. The rendering part takes the idea from 3D neural rendering where a shared NeRF function is used to represent all objects excluding the background. Rendering is performed by querying NeRF with 3D location, 2D view direction as well as object latent to get object color value and occupancy value. Object rendering is composed into scene rendering according to location derived at the inference stage and weighted by occupancy value.

To sum up, the paper interprets a 2D visual scene with 3D object-centric representation. With the existing 2D object-centric scene segmentation method and 3D neural rendering approach, it achieves comparable segmentation performance and derives manipulable object representation.
main_review: Strengths:

1. It is interesting to use NeRFs to represent objects in a scene, which encourages the understanding of 2D visual scene taking account into its 3D structure.
2. Explicitly modeling the 3D structure of 2D scene may help better deal with occlusion, ambiguity from similar appearance which can be a big challenge in 2D scene understanding.
3. It looks great to show how learned object features can be used for downstream tasks.

Weaknesses:
1. The technical novelty is somewhat limited. The paper mainly combines the Slot-attention and NeRF. Personally, it is hard to identify the exciting point(s). 

2. The motivation of this work seems not well justified. It is claimed that the model is proposed to infer 3D representation for objects that are interpretable and amenable. But the experiments presented with such 3D representation cannot show its superiority and it is
hard to see the necessity of 3D object representation in this case considering the task presented can also be fulfilled with 2D object representation. In terms of the interpretability and manipulability of latent code, this paper presents objects removal/addition, object scaling and object translation. All of the above operations can actually be done just by explicitly modeling 2D object pose such as AIR (NIPS'16 paper). In terms of scene understanding in the form of object segmentation, this model cannot beat Slot Attention. In terms of downstream tasks, this model is not compared with other object-centric models. Thus, based on your current experiments, the motivation for learning such 3D representation for objects is not well justified.

3. Is it appropriate to use NeRF? If I understand correctly, NeRF learns the radiance field of a scene with 2D images rendered from different viewpoints. This model tries to recover the object radiance field from a single view of the objects. But the model training
objective can actually be fulfilled by just learning a single-view of the scene since there is only one image for each scene to generate reconstruction loss.

4. From my understanding, the 3D rendering model in this paper seems more similar to a decoder in VAE-wise architecture. It could be more convincing to provide some novel view synthesis (not novel scene) to claim it as a NeRF. For example, you may consider
object rotation.

5. All experiment datasets are a bit too simple. It is unclear how good/bad the proposed model will perform on realistic 3D scenes such as LLFF dataset or ScanNet. Otherwise, it is hard to see the real impact of this proposed method.

summary_of_the_review: Look forward to the response from authors.
review_title: null

Meta-review: The paper proposes an approach for learning a decomposition of a scene into 3D objects using single images without pose annotations as training data. The model is based on Slot Attention and NeRF. Results are demonstrated on CLEVR and its variants. 

The reviewers point out that the method is reasonable and the paper is quite good, but even after considering the authors' feedback agree that the paper is not ready for acceptance. In particular, the key concern is around experimental evaluation - that it is performed on one dataset (and variants thereof) and that the evaluation of the 3D properties of the model is not sufficiently convincing: it does not outperform 2D object learning methods on segmentation and is not compared to those on "snitch localization".

Overall, this is a reasonable paper, and the results are promising but somewhat inconclusive, so I recommend rejection at this point, but encourage the authors to improve the paper and resubmit to a different venue.

(One remark. The paper makes a point of not using any annotation. It is technically true, but in practice on CLEVR unsupervised segmentation works so well that it's basically as if segmentation masks were provided. If the authors could demonstrate that their method - possibly with provided coarse segmentation masks - works on more complex datasets, it would be a nice additional experiment)
Decision: Reject