Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new model for unsupervised object-centric 3D scene understanding, called MORF. Unlike previous methods, it leverages 2D image segmentation masks from the pretrained optical flow method EISEN. These are used to initialized a slot based representation augmented by pixel features (as in PixelNeRF), which are used to condition NeRF decoders for objects and backgrounds. On a novel synthetic 3D dataset called Playroom, the model yields better reconstructions and 3D geometry results when compared to prior work. Improved scene editing capabilities are demonstrated, as well.
strength_and_weaknesses: Strengths:
 - The paper addresses a relevant and interesting problem, namely, unsupervised object-centric 3D scene understanding.
 - It is well written and easy to follow.
 - The idea of incorporating motion cues for better segmentations makes sense and has not been explored for 3D models so far.
 - MORF convincingly outperforms uORF on the new dataset.

Weaknesses:
 - The characterization of prior datasets ("They only demonstrate simplistic scenes with a single object category, and the objects in the scene are in uniform colors.") is not accurate. While they are synthetic and might indeed by called simplistic, e.g. Multishapenet from ObSuRF and MSN-hard from OSRT use objects from multiple ShapeNet categories, and their standard, multi-colored textures.
 - Consequently, I disagree that the new dataset represents the jump in complexity that the paper suggests it does. To me, it appears of a similar level of complexity as ObSuRF's Multishapenet, and much simpler than OSRT's MSN-hard.
 - Increased performance comes at the cost of requiring video data for training, which the baseline models don't.
 - The fact that the processing of video data is outsourced to an existing 2D model decreases MORF's technical novelty, and doesn't make use of the full potential of combining 3D with video observations.
 - Given the emphasis on novel view synthesis in the evaluation, the state of the art model SRT (Sajjadi et al., 2021) should be included as a baseline, even though it will not yield segmentations or 3D geometry in its default configuration.
 - Some additional information could be added for clarity (see below).
clarity,_quality,_novelty_and_reproducibility: __Clarity__: In general, the paper is well written and easy to follow. Some additional data would be useful to provide to make it easier to evaluate qualitative results:
 - Dataset characteristics such as resolution, number of video frames, etc.
 - Training time / computational cost of EISEN and MORF.
 - Videos from the dataset, as well as EISEN segmentation results compared with ground truth segmentations.
 - Videos showing novel view synthesis and segmentation results.
 - Extracted 3D meshes.

__Quality__: The experiments are setup in a reasonable way, although the inclusion of an SRT baseline as discussed above would be nice. I think not comparing to ObSuRF or OSRT is acceptable, as they operate in slightly different settings. Even though this is largely provided by EISEN, I think a segmentation comparison between uORF, EISEN, and MORF would still be useful, as the geometric scores blend together segmentation and 3D reconstruction quality.

__Novelty__: The fact that MORF mainly leverages cues from an existing, pretrained model reduces its novelty. That said, it is a first step towards the challenging setting of 3D video data.

__Reproducibility__: The authors have promised to provide code and data, which goes a long way for reproducibility. Neither has been provided for review, however. Taking the paper by itself, many implementation details are missing, such as the architectures of the image encoder, or the NeRF MLPs.

summary_of_the_review: Overall, the paper represents an interesting first attempt at utilizing both video and 3D cues for better unsupervised scene understanding. While I believe its novelty is somewhat overstated, the experiments are competently executed. I am therefore leaning towards acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends previous work on 3d scene decomposition and reconstruction from a single rgb image. Being able to reconstruct the 3d structure of a scene from a single rgb enables a host of new functionality such as re-rendering multiple views of the scene, editing the scene, and potentially generating images for training 2d and 3d inference models. This work is novel in that it generalizes previous work to arbitrary "movable" objects rather than relying on a supervised model trained on a fixed set of objects.  This is a step function improvement in capability enabled by this work. 
strength_and_weaknesses: The strengths of this paper are as follows:
* Allows for scene rendering from a single image and scene editing. 
* Enables a new level of capability by generalizing previous work to handle multiple and arbitrary classes of objects in a scene rather than specific objects trained in a supervised fashion.
* Convincing quantitative and qualitative results demonstrating that the approach works as expected. 


The weaknesses of this approach are as follows:
* All results are demonstrated only on synthetic data.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, is high quality, and the authors state that they will release code and data on acceptance, so reproducibility  should not be a problem. 

This work does seem to be a fairly straightforward combination of  EISEN (Chen et al, 2022) for detecting 2d masks of arbitrary objects and uORF (Yu et al, 2022) for object-based scene reconstruction.  Their Loss function is almost identical to Yu with the addition of the adversarial loss component.  There is no ablation to indicate how important that additional loss is so it's hard to know how significant that addition is.  I would love to see this added in future versions. 

The similarity to the work of Yu is not called out very clearly. They mention Yu in the related work section and is one of the primary baseline methods used, but during the discussion of the method implementation there is little indication of the similarity to Yu, and it is left to the reader to put this together. 


summary_of_the_review: This is a straightforward combination of two techniques that enables a step-function capability in 3d scene reconstruction from rgb images. I think this is a solid contribution and worth accepting at ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Movable Object Radiance Fields, by using the EISEN method to generate object masks and processing object and background separately. Comparing with uORF and PixelNeRF methods on three self-generated datasets, the authors show that the proposed method can extract accurate object geometry and has higher performance in scene reconstruction and editing.
strength_and_weaknesses: Strength:
This paper proposes a Movable Object Radiance Fields, which separate 3D scenes into objects and background before reconstruction. The authors claim that this method can better handle complex and diverse multi-object 3D scenes.


Weakness:
1.The overall structure takes advantage from many advanced algorithms, e.g.. object mask generation, object latent code computation, compositional rendering. I think the authors should better clarify the novelty and unique contributions of their method. 

2.The authors carry out their experiments only on self-generated datasets. I think that for better comparison, they should test on more public datasets, e.g. the ShapeNet dataset, the DTU dataset, etc.

3.The authors use a long description to introduce the EISEN segmentation method. I think it would be better if they explain why EISEN, not other methods are chosen to for mask generation in the proposed method. Besides, they should provide the evaluation results for segmentation quality of EISEN; maybe even apply some more segmentation methods to show the influence of object masks (right now they only compare EISEN with GT masks without quantitative mask qualities).
clarity,_quality,_novelty_and_reproducibility: The ideas are clear and reproducible, with limited novelty.
summary_of_the_review: The proposed algorithm is not well compared and demonstrated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new framework for 3D object centric learning by leveraging the 2D object mask extracted from the motion information. The proposed approach then trains separate object and background nerf using different network structure. Experiments on data with more complex texture demonstrates the performance of the method.
strength_and_weaknesses: #Strength 
+ The paper leverages the motion information for 3D object centric learning. It can handle images with complex textures

+ The object nerf and background nerf are trained based on global object feature and pixel-wise features.

 
#Weakness
+ Novelty of the paper. The proposed approach highly depends on the 2D segmentation mask which provides significant information for 3D object centric learning. Moreover, the 2D object masks are obtained from an existing work and the formulation for the object compositional nerf is not new [R1], which renders the proposed method lacks novelty. Please refer to the following missing reference for the compositional nerf [R1, R2].

 +  [R1] Unsupervised Discovery of Object Radiance Fields, ICLR2022
 + [R2] Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering, ICCV2021.

+ The 2D mask segmentation framework highly depends on the quality of the optical flow. Then it cannot handle the images of pure color.

+ Eq. (4). Should ${\bf c}$ be ${\bf c}_i$? 

+ It would be great to explain why results from the proposed approach cannot recover sharp boundaries and the reconstructed objects are larger than the real one (see Fig. 4)

+ It is hard to interpret the scene editing results. It looks like all views are different from the ones shown on the top? It makes it difficult for the reader to understand whether the images are rendered from changed objects’ positions or just the changed camera view point.

clarity,_quality,_novelty_and_reproducibility: + The paper is well-written in general. The reviewer is concerned with the novelty of the paper by considering it took a few components from existing works.

+ It would be great to explain the advantage of using the object level feature.

+ The method is reproducible.

summary_of_the_review: This paper tackles a challenging problem, unsupervised 3D object-centric learning. The proposed framework is convincing. However, it largely depends on the 2D mask information obtained from a 2D image segmentation framework and the object compositional nerf is not new. Thus, the reviewer is concerned that there is not sufficient novelty in the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new model for unsupervised object-centric 3D scene understanding, called MORF. Unlike previous methods, it leverages 2D image segmentation masks from the pretrained optical flow method EISEN. These are used to initialized a slot based representation augmented by pixel features (as in PixelNeRF), which are used to condition NeRF decoders for objects and backgrounds. On a novel synthetic 3D dataset called Playroom, the model yields better reconstructions and 3D geometry results when compared to prior work. Improved scene editing capabilities are demonstrated, as well.

The paper addresses a relevant and interesting problem. It is well-written and easy to follow. The idea of incorporating motion cues for better segmentations makes sense and has not been explored for 3D models so far. MORF convincingly outperforms uORF on the new dataset.

The characterization of prior datasets is not accurate. The new dataset represents the jump in complexity that the paper suggests it does. Other state-of-the-art models should be included as a baseline. More information could be added for clarity. All results are demonstrated only on synthetic data. Clarify what novelty and contributions come from this article and not prior work. Use public datasets. The novelty of the paper. It is hard to interpret the scene editing results.

The authors have done a good job answering to the reviewers. However, these still are not fully convinced about the novelty.
justification_for_why_not_higher_score: Not enough novelty.
Decision: Reject