Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method to synthesize novel views of 3D scenes while inferring a decomposition of the scene into multiple objects.  The compositional reasoning is achieved by a slot-based encoder so that there's no need for specific supervision on the object categories. The authors also contribute with a novel loss function that exploits depth training data to improve the sampling strategy and therefore increasing training speed as well as lowering the reconstruction error. The results out-perform the state-of-the art, and are validated against 2D and 3D datasets.
main_review: The paper is well written, is pleasant to read and the formulation seems sound. The authors clearly justify their decisions and overall their work provide valuable insight about the problems of competing approaches in the literature. The experimental validation seems well designed and the results out-perform the state of the art for both 2D and 3D datasets.

Regarding the novelty of the work, there are two main aspects: First, the unsupervised decomposition of the scene, which is somehow incremental, given that it's achieved by applying the slot-based approach of [Locatello et al. 2020] to NeRFs, and also very related to [Yu et al. 2021]. Second, the training speed-up using depth training data, which seems to be the most significant contribution. However, this was also explored in a recent work by Kangle Deng et al. "Depth-supervised NeRF: Fewer Views and Faster Training for Free". 

I do understand that since this work was published in arxiv in July 2021, according to ICLR guidelines the authors are not required to compare their work against it but given that the most significant contribution is so similar, I would find necessary to reference the paper and briefly mention it in the related work. 

The authors comment on the fact that it's possible that some of the reconstruction error gains are due to the additional (depth) supervision. This renders the comparison a bit unfair given that the present work is using additional training data. I miss further discussion, in a similar fashion as the ablation experiment with the overlap loss, but focusing on the depth training data. 

summary_of_the_review: I would recommend this paper for acceptance as long as no other reviewers have major concerns. The paper conveys a good contribution. It's quite unfortunate that those two recent works (Kangle Deng and Yu et al. , both in July 2021) clash significantly with the two main novelties of this work, but as stated above, those papers are too close to the submission deadline to be considered double submission. In any case, I find the citation of the work of Kangle Deng. et al. a requirement for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors presented a method to infer a neural radiance field per object in a scene from a single input view. 
uses  Object-centric learning with slot attention

The proposed architecture takes the shape of an autoencoder. The encoder side takes a single image and pose as inputs and generates N latent codes for each object in the scene following  Object-centric learning with slot attention (NeurIPS2020). The latent codes condition shared NeRF decoders to define the geometry and appearance of each object. The full scene is reconstructed by superposing the multiple Radiance Fields.

Furthermore, to reduce the computational complexity of NeRFs, the authors propose to use GT depth maps to avoid full ray marching.
main_review: I found the paper a bit hard to follow, as it heavily relies on previous works. However, I think successfully builds on previous works and shows promising results on the used synthetic data.

Strengths:
* Using depth to reduce the number of samples considerably is an intuitive way to reduce the computational complexity of training NeRFs.
* The proposed method considerably outperforms (quantitatively) the previous methods under comparison in Tables 1 and 2.

Weaknesses:
* The segmentation part seems to be already proposed in  "Object-centric learning with slot attention (Neurips2020)". What is the difference?
* The conclusion section could be improved to be more specific on the main paper achievements/observations.
* Visual comparison against the previous methods in Table 1 and/or 2 are missing.
* No results on real-world data are provided.
* Similarly, the results seem to be only good on the CLEVR dataset which contains very simple geometries. For the ShapeNet dataset, the results seem to be rather blurred.
summary_of_the_review: The results shown in this work seem promising and advance towards single view 3D scene understanding. However, due to the mentioned weaknesses in the main review, I am leaning towards a "marginally below the acceptance threshold" score.

Update:

The authors have addressed the comments in my initial review and showed improved results on the Shapenet dataset. Therefore I am updating my recommendation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about unsupervised segmentation both of 3D volumes and of images. The idea is to learn a set of Neural Rendering Fields (NeRFs) which explain a given image. To reduce computationally effort, the authors assume RGB-D data and show how that reduces the computational burden by two magnitudes. The proposed algorithm can learn segmentation masks and a set of NeRFs that generate each individual object and the background.
main_review: I like the idea to use more information allows to use a more efficient algorithm. The topic is out of my field yet the derivations seem theoretically sound. The experimental evaluations (using simulated data) demonstrate that the algorithm works, at least on very simple data.

I have a few questions and remarks:

- What does "Eq (3) is renormalized" mean? Please explain that procedure.

- "a significant chance that thin, high-density volumes are missed": Standard hardware for capturing RGB-D data likely does not capture thin, high-density volumes either, thus the proposed approach would not be successful for such data. This somewhat implicit argument (the proposed approach can solve thin, high-density volumes) should be removed, thus. 

- Can the authors drop a few words about the derivation of log p?

- Eq (4): Why are 50% of the samples taken from the last 2% of the ray? What happens is the ray is sampled differently?

- maximizing Eq 4 under a Gaussian distribution: Is this assumption justified? Can it be violated? If so, which attributes would such a part of a volume have?

- How is the joint log-likelihood p(t,C) defined? Why are only 2 evaluations of the NeRF necessary?

- What do the authors mean by "we use the code to shift and scale the activations at each hidden layer"?

- By Fig 1, I assume that the number of objects is provided for each image? Stated a bit differently, it seems that differently many slots are used for the input images? That reduces the claim in the title about the unsupervised segmentation since providing the number of objects can be considered as a weak supervision signal.


summary_of_the_review: A major disadvantage of using NeRFs is their need for very expensive GPUs. I like that the authors address this issue and propose a solution. The price that one has to pay for that (RGBD instead of RGB, providing the number of objects) can be justified, often. Furthermore, I suppose that this paper will trigger follow-up works which improve the weaknesses of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an array of techniques for solving the problem of decomposing a single image of a scene into a set of NeRFs, each of which represents an object in the scene. The decomposition requires accurate estimation of scene depth and the segmentation of the scene into objects. To achieve this, the authors propose an enhanced version of NeRF that operates on RGB-D images where its rendering only requires two evaluations per each camera ray. The authors show that the proposed method works effectively on CLEVR-3D and a custom built dataset named MultiShapeNet. 

main_review: Strength:

- Decomposing a scene into objects in 3D is a challenging problem and solving this could potentially be useful for the area of explainable AI. 

- The probabilistic interpretation of ray marching in Section 3.2 is interesting and somewhat novel in the context of NeRF, but this piece of knowledge is well understood in the physically based rendering community.  

Weakness:
- The problem authors seek to address is not defined clearly. The title and the
main concept of the paper are inconsistent. According to the title, a new 3D scene segmentation method is proposed. However, the input of the pipeline is 2D images which is prone to be 2D instance segmentation.
- Lack of novelty. 
Using depth to facilitate training is proposed[1].
[1]Depth-supervised NeRF: Fewer Views and Faster Training for Free
Composing NeRFs has been explored before, as acknowledged by the authors in the citations. This paper arrives at similar formulations but from the probabilistic point of view. 

- The presentation sometimes gets complicated with too long background information (Section 3.1, 3.2, 3.3) that causes some confusions. 
summary_of_the_review: Summary
The authors use an encoder network to infer a set of slots for each object contained in the input image and then these features are used in Nerf to generate novel views. However, it lacks the motivation of leveraging nerf lack. The authors must make it clear. In addition, it lacks experiments to verify the proposed methods. Evaluation of advanced 3D object dataset such as Google Scanned Objects is necessary. Overall, the paper should be rejected.

Questions to authors

1) The paper seems to include some contributions that do not directly solve the problem. For example, can the decomposition work with traditional NeRF and ray marching instead of the proposed ray evaluation? 

2) Section 3.3 mentions that the ray marching in NeRF is biased, which is true and a well known phenomenon in volume rendering. 
Then the authors claimed that the proposed ray evaluation is unbiased. Could you provide a formal proof? 

3) The comparison does not include results that evaluates segmentation and depth separately using state-of-the-art methods for each aspect. How does this method fare compared to those SOTAs? 

4) Ablation study is lacking. Overall I can see that the proposed method outperforms a few baselines but there is not much analysis into the proposed method. Statistics about running time should be shown too as the proposed ray evaluation could be faster than traditional ray marching in NeRF. 
For contribution in Section 3.3., output quality compared to NeRF should also be shown. 

5) The experiments are done with synthetic datasets only. Since the proposed method is based on NeRF, how well does it work with real photographs? 

Detailed comments

- Abstract: the motivation of segmentation is not clear. Infering a set of NeRFs, one for each object, from a single input image is a good approach, but the connection to segmentation of the scene is left unexplained. 

- The derivations are sometimes quite lengthy and sometimes the motivation is not so clear. For example, Section 3.1, 3.2, 3.3 only provides background, and then Section 3.4 (which appears quite late in the text) actually tackles the problem. 

- The experiments are only reported with quantitative results. Having some visualizations of the segmentation and depth and some novel view renderings in the results would be nice. 

- When comparing segmentation results, please add IoU metric. 


Rating

The paper sounds weak in the sense that some of its contributions are on improving NeRFs while the decomposition part is rather marginal. The presentation is sometimes misleading. There are a few concerns about the experiments and baselines as well. Therefore, my current rating is below the acceptance threshold. I invite the authors to try considering the questions as commented.
review_title: null

Meta-review: The paper develops a method for decomposing 3D scenes into objects by coupling NeRF decoders to representations produced by a slot-based encoder.  After the discussion phase, reviewer ratings are mixed with three on either side of above/below threshold, and one higher (but low confidence) accept score.

Drawbacks include limited novelty, as stated by Reviewer 8UAh: "the unsupervised decomposition of the scene, which is somehow incremental, given that it's achieved by applying the slot-based approach of [Locatello et al. 2020] to NeRFs".  Reviewer bAmB likewise mentions this issue.  Reviewer VrrK: "some of its contributions are on improving NeRFs while the decomposition part is rather marginal". Reviewer VrrK also raises concerns about lack of experiments on real data: "Since the proposed method is based on NeRF, how well does it work with real photographs?"

The AC agrees with the marginal rating of the reviewers and is particularly concerned with overall novelty of the proposed pipeline and question of applicability beyond simulated data.  More work seems required to solidify an experimental case on real images.
Decision: Reject