Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel neural representation of an given image, which decomposes K object instances from background. So various tasks such as rerendering, rearrangement etc. 

The learning process is first sampling K centers, and represent each object as a learnable hidden variable z,   an gaussian based soft. k-means styple clustering is then performed afterwards. The differences here are 1)  for sampling the centers, there are  learnt forground and background priors, which can benefit the initial state in this learning process.  2)  updating the centers z with a learnable GRU rathor than simple mean pooling, which I believe it have more flexible representations for the cluster. Finally,  the object clusters are discoverred. In this process. 

The author evaluted with 3 self created datasets and show several reasonable results by performing mentioned tasks such as 3D segmentation, rearrangement etc. 

main_review: strengths:

The overall direction is promising, and factorize the scene representation is indeed an important issue to study. 
The technique proposed is sound overall with soft-kmeans like strategy to generate corresponding features in an unsupervised manner, through the GRU probably break the theoretical guarantee of convergence. 

Weak: 

The overall results looks more in a concept proof, the objects in all test datasets are relatively simple having uniformed color. Feel it can hardly work in real scene senario, as shown in GIRAFF paper.  Under these senarios, the segmented results and rearranged results lose many object details yielding blurry or incorrect renderring.  I feel there should be more improvement over these issues. 

The overall concept is fairly close to GIRAFF and major difference could be the training scheme inference from a single image or multiple. I would like to see there could be additional techniqual improvement especially some high resolution representations. Or improvement of architectures in order to support better quality. 

some questiions: 
Does the algorithm always obtains a reasonable representation w.r.t different initial sampled centers ? 




summary_of_the_review: This paper points out a good direction to dive into unsupervised learning of compositional scene representations. However, technique strongness and novelty may need to be further improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper utilized the powerful NeRF. The authors present a new approach to learn the scene arrangement in an unsupervised way. The training is performed on unlabeled datasets of similar objects in different arrangements. The inference requires only one RGB image as input, and can correctly deduce the arrangement and the 3D geometry of the objects. The authors showed two supporting technical contributions to the system: (1) splitting background and foreground objects leads to better results, (2) a coarse-to-fine training to alleviate the space and time needs. The authors showed success on three synthetic datasets and various applications.
main_review: The authors presented a novel idea. The system is engineered well and the authors have shown success on three synthetic datasets and various applications.
My concerns are as follows.
1. All experiments are conducted on synthetic datasets. Both training and testing use the same set of object shapes - only the arrangements of the objects are different. It is not clear how this approach can generalize to real world scenes. As in the real world, lots of objects have not been seen in the training set. An ablation study that adds unseen shapes into the testing scenes can be very informative. Additionally, a demonstration of the approach on real world scenes would be a strong result to show in the paper (either it is negative or positive).
2. How is the number of the foreground objects decided? Does it have a strong impact in the results?

Edit: The authors addressed my concerns in their revision. In particular, the authors showed additional results on a real world image. As expected, the rendering is not as good as on synthetic data. However, I do not think this overshadows the contribution of this paper - instead, it shows the value as well as limitation of the proposed method, and can inspire future work.
summary_of_the_review: The authors presented a novel idea. The system is engineered well and the authors have shown success on three synthetic datasets and various applications. However, all evaluation and experiments are done on synthetic datasets with the same set of objects and background. Thus it is unclear how this approach can generalize to solve real world problems.

Edit: The authors addressed my concerns in their revision. I think it is a good paper and should be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces an interesting new research direction of factorized, 3D consistent neural representations. In particular, it proposes to combine slot-attention mechanisms with conditional neural radiance fields to segment and render novel views of a scene from a single input view. The authors also address one apparent shortcoming in the slot-attention paper: the background and foreground object latent codes are sampled from the same distribution, leading to breakdowns on scenes with complicated backgrounds. The paper proposes to learn two disjoint distributions, one for the background and one for the foreground, to alleviate this issue.
main_review: ##################################################################
# Pros
1. The authors address a significant new problem: modeling 3D scenes as a disjoint set of objects that can be combined and rendered for novel view synthesis. Their model can also be learned from only 2D data.

2. The proposed method uses state-of-the-art techniques to achieve its goal. Namely, it uses slot-attention mechanisms (NeurIPS 2020) and Neural Radiance Fields (ECCV 2020) and combines them to address a new problem.

3. Treating background and foreground latent vectors as being drawn from two separately learnable distributions addresses one of the significant drawbacks of slot attention.

4. I appreciated the authors mentioning that concurrent work by Stelzner et al. (2021) addresses the same issue and differentiates this paper appropriately. 

5. The proposed method is overall technically sound, and code is provided, which will help with reproducibility. Also, the authors do an excellent job at mentioning all the hyper-parameters and model architecture details as far as I can see.

6. The paper provides comprehensive experimental evaluations, both quantitative and qualitative.

####################################################################
# Negatives / Questions
1. It would have been great to compare this work to the concurrent work of Stelzner et al. (2021). I believe that this would help the community to put the two concurrent submissions into context. That said, I do recognize that the work of Stelzner et al. (2021) has not been published and that code/data for their approach is not publicly available, making comparisons extremely difficult and should therefore not be a requirement for publication of this work. I hope that future work in this direction will pick up this issue.

2. Section 3.3, Coarse to fine training. I agree that rendering images with the volumetric rendering framework proposed in NeRF requires many evaluations per ray, so reducing the number of rays sampled during training makes sense. One detail that is missing in this section is how many samples per ray are used? Moreover, do the authors still use the two networks for ray evaluation as NeRF (coarse and fine ones)? If not, why not? Second: In your approach, you sample random patches during the fine training stage and downsample the images during the coarse training stage. In this paper: "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis," the authors propose a different sampling strategy, which does not require downsampling. Would this strategy perform better? Worse?

3. Section 4.1, Segmentation Experiment Results. It would help to discuss the shortcomings of slot-attention, namely, why the method fails on the segmentation task. My guess why uORF works better would be due to the two disjoint latent spaces for foreground and background? I think this should be highlighted here.

4. Section 4.3, Scene Design and Editing Setup. I did not understand how the setup for modifying the foreground object pose/appearance works. I think you can switch the latent embeddings for background, as you have a one-to-one mapping, but this is not the case for the foreground objects. Could you explain this process in more detail?

5. Appendix B2, Coordinate Space. Here you mention that you use a foreground box to encourage disentanglement of foreground and background slots. How does this foreground box work, what is its influence on the final result? Mentioning this (maybe) crucial detail only in the appendix is not sufficient in my opinion and should be better explained in the main text.

summary_of_the_review: I vote to accept this paper for publication at ICLR 2022. I like the idea of modeling scenes as a combination of disjoint objects, which can be added, removed, modified, and recombined to form new scenes. I also think the paper is well written, well-motivated, and provides extensive experiments. In my opinion, the paper adds to the literature on neural scene representation/decomposition and is interesting to the community. I have some minor suggestions and questions (see above), which I hope the authors clarify during the rebuttal.
review_title: null

Meta-review: This paper develops a method for decomposing scenes into object-specific neural radiance fields.  After the discussion phase, two reviewers support acceptance.  Empirical results on multiple synthetic datasets and benchmarks appear convincing; the rebuttal also added an initial demonstration of generalization to real images.
Decision: Accept (Poster)