Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends on the power NeRF, by introducing an additional vector to present the object code for each 3D point. The authors designed a few loss functions and showed that with these loss functions the object code for the 3D points can be directly optimized from 2D image annotations. The authors then used the learnt object code to translate/rotate/scale the objects and showed promising experimental results.
strength_and_weaknesses: + The idea is novel. The design of the object code and the loss functions is simple and elegant.
+ The authors have done extensive experiments on a few different datasets and showed superior results comparing with MaskRCNN
+ This paper is a resubmission of an ECCV 2022 submission. The comments in the review and rebuttal have been addressed quite well in this submission.

- The authors mainly compared their approach with only one existing approach (MaskRCNN). However, I do agree with the authors' explanation in section 4.2 that most relevant work focuses on similar but different problems thus it is hard to do a fair comparison with these works.
- In all manipulation results, the proposed method still has a ghost boundary along the model pre-manipulation. Can you discuss this limitation?

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read. However there are a few typos, just to name a few:
Section 1, 3rd paragraph: simple amodel -> simple model
Section 1, 4th paragraph: with a ability to clearly -> with an ability to clearly
Section 3, 1st paragraph: this villina NeRF -> this vanilla NeRF
Section 3, 1st paragraph: together with a object -> together with an object

summary_of_the_review: This paper presents a good idea and executes it well. The design of the object code vector and the loss functions is neat. The authors show superior accuracy compared with MaskRCNN. They further used this to demonstrate an application of manipulating the objects. This paper is a resubmission of an ECCV 2022 submission. The authors addressed lots of comments from the previous review cycle, and have made it a good submission for ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This submission proposed an extension of NeRF called DM-NeRF, which can segment individual objects and manipulate them (translation, rotation, deformation, etc.). To supervise object segmentation, the authors use their algorithm to generate multiple 2D object code predictions and match them with ground truth object labels using the Hungarian algorithm. To manipulate an object, they perform the inverse operation on sample points along a ray. The empirical results show high-quality novel view synthesis and better decomposition than Mask-RCNN.
strength_and_weaknesses: - The task studied in this work is important. As the authors mentioned, many cutting-edge tasks may require obtaining and manipulating individual 3D objects in a 3D scene.
- The object code supervision proposed by the authors is natural and generates good results. As the authors mentioned, supervising the object code prediction is challenging as the ordering of the objects is not determined. The multiple views of 2D object code with the Hungarian algorithm is a good solution to it (idea originated from 3D-BoNet), and it produces better results than Mask-RCNN.
- The object manipulator produced reasonable results but has space for improvement. Empirically, the manipulator produced artifacts (Figure 7, first line). At the end of the video “DM-NeRF_2D.mp4”, when the table is being rotated, they are some very unrealistic artifacts as well as unrealistic light.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clearly written, though the source code is not provided. This work is an extension of NeRF, which also predicts an object code for segmentation. The major novelties are segmentation supervision and the inverse query algorithm for object manipulation.
summary_of_the_review: The authors proposed to extend NeRF object instance segmentation, providing a way to supervise the object code prediction which produced good results (better than Mask-RCNN). They also proposed an object manipulator which supports translating/rotating/scaling an object but generates artifacts and did not consider lighting conditions, which is my major concern.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a novel formulation based on Neural Radiance Fields (NERFs) to jointly reconstruc, semantically decompose, manipulate and render complex scenes in a unified pipeline.
strength_and_weaknesses: ** Strengths:

- (1) Readability.

In its current state, the paper is very well written. The main key ideas are well explained, motivated, justified and articulated.

- (2) The relative positioning is very clearly and explicitely established.

This includes multiple discussions, in particular the differential wrt other existing object-centric NERF prior work.

- (3) The overall maturity of the proposed package.

- (4) The Related Work discussion is strong, overall.

Its structure, populated references are pertinent and up to date, while the level of insights is a very good trade-off to make an efficient use of the text real estate.

- (5) The performance of the proposed contribution, despite the relatively competitive field. 

** Weaknesses:

- (1) The experimental quantitative evaluation lacks perspective.

This applies throughout and hurts the contributions: 
(i) decomposition contribution: Mask-RCNN (He 2017) is now considered a standard baseline rather than a strong state-of-the-art reference. In turn, the near perfect performance of the proposed method on the proposed dataset raises questions regarding the pertinence of the selected scenes to evaluate altogether. Having a more compelling baseline or multiple ones could help the reader accept the proposed contribution at the method level, near perfect. In absence of the aforementioned, it is still difficult to accept as is.

(ii) Object Manipulation: there should be a better justification for a complete lack of a baseline, outside the ablative study as an experimental contribution.


(iii) The contributed dataset: the proof of concept applications to validate its usefullness and quality is strongly tied to (i) above, which makes it slightly problematic as it currently stands.


- (2) How much does it cost? (Conceptual discussion)

There are missing information about typical runtimes, resource usage and specificities, in particular in contrast to the closest related NERF-based derivatives out there.
clarity,_quality,_novelty_and_reproducibility: Most of it is on point, but there is one missing link between the 3 layers of contributions, as stated above, which currently makes it stlightly of a tough sell without this missing justification or experimental addition.
summary_of_the_review: 
There are multiple positives to this one. However, the main missing bits hurt all three levels of contributions made by the submitted paper as mentioned above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of 3D object segmentation and manipulation from images with instance segmentations. It uses NeRF and ask it to further produce an object label per 3D point. They introduce a Hungarian matching based loss to supervise the object label fields with 2D instance segmentations (although that part wasn't clear in the paper). They show better results than the 2D mask-rcnn baseline in terms of 2d segmentation.
strength_and_weaknesses: Strength
- The problem of 3D object segmentation from images is an important problem, and this paper takes an interesting step to solve it.
- Novelty. The Hungarian matching-based loss seems to be an interesting solution to handle the 2D instance label mismatch during loss evaluation. However, the paper did not provide much detail. For instance, what is the cost function when matching the rendered labels to the ground-truth. When it may fail?

Weakness
- Clarity. My major concern is on writing and clarity. The method is hard to keep track of, and I don't think readers can easily understand how it work. To improve the writing, the authors may want to improve the figures, use precise notation for losses, and put adequate emphasis on certain parts. See minor comments for more.
- Experiments. I found the comparison against over-fitted mask-rcnn to be not informative. Assuming the ground-truth segmentation is perfect, mask-rcnn fine-tuned per scene could easily overfit to the ground-truth labels and produce high accuracy. The fact it did not produce high accuracy made me think it is not well-trained. I can only see this experiment to be meaningful if the input segmentation is noisy, and the paper aims to clean it up, while mask-rcnn is not able to do so. 
- Generalization to real data and other setups. The method seemed to be trained on synthetic data with good instance segmentation. If we were to apply it to real world images, we need to answer how sensitive is the method to bad 2D instance segmentations. A related question is how to handle results of existing detectors (for instance Mask-RCNN only gives instance segmentation but not the background and stuff classes). Does it work with panoptic segmentation?

Minor comments
- Fig 1: object manipulation block is left empty, is this intended?                      
- Fig 2: losses appeared but not discussed or referred to. Ideally, by looking at the figure, readers would be able to understand the gist of the method.
- Fig 3: does this corresponds to l-2d-obj? Please make it clear in the caption.
- Eq 4: $\delta_k$ is not defined. Is it the same as $d_k$? If $\delta_k$ is the depth, Eq 4 shouldn’t be called surface distance since it’s measuring the distance to the closest surface along the ray instead of the closest surface in the scene.                                                                                            
- Eq 5: how does the indicator function work here? If $d-\Delta d-d_k =0$, value is one?
- Eq 7: The name of $l_{3d-obj}$ is slightly confusing as it optimizes the emptiness of the object label, but the name indicates it has similar functionality as $l_{2d-obj}$, which optimizes the object label.
clarity,_quality,_novelty_and_reproducibility: See strength and weaknesses. I don't think it is easily reproducible given the unclear notation and loss functions (mentioned in minors).
summary_of_the_review: Overall, the problem setup is novel and the solution is new. However, the method is not clearly described and the comparison to baseline is not well-designed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the structural decomposition of 3D objects from 2D views. 

The idea is novel; however, at the submission time, some critical experiments are missing. Therefore, the paper received mixed scores. The authors provided a strong rebuttal, including experiment results submitted a bit behind the deadline of Stage 1 of the discussion period. After a few rounds of discussions among reviewers and the area chair, the committee recommends to accept the submission.ted. 
justification_for_why_not_higher_score: See the Summary of AC-reviewer meeting for the debates during the discussion phase. There are some issues caused by the author side in reviewing the work. 
justification_for_why_not_lower_score: The work provides a value idea with a good amount of experiments.
Decision: Accept: poster