Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper propose an unsupervised multi-view object segmentation method leveraging on neural radiance field. The key motivation is that a good segmentation should  maximise the dis-similarity between content within each individual mask. Correspondingly, three key modules including radiance propagation, bi-directional photometric loss and EM-based post-processing are adopted to decompose a scene into different regions/objects. This paper presents promising qualitative results on object-centric scenes and synthetic multi-objects scenes under multiple observations. 
strengths_and_weaknesses: Strengths

1. The paper is well motivated and the pipeline design follows the idea of making prediction  of one part from others difficult.

2. The evaluation and ablations are adequate on single-object scenes and multi-object scenes(synthetic)scenes. As far as I can tell this is the first multi-object segmentation framework in a purely unsupervised manner on complex scenes.

3. Though there are some missing details (mentioned in the weakness part), the overall writing is good.



Weaknesses


1. It is not clear to me what equation (4) means and helps the optimisation? In addition, \hat{l} is not mentioned in the remaining text.

2. The object-level NeRF is driven to predict average colour within masks across the batch, the design choice seems a bit ad-hoc. 
What about predicting other related values to object regions like the maximum or minimum or other strategies? 
As to objects with complex textures or various colours, I assume even the average colour may also not well represent the object appearance. Objects within the paper owns similar textures or colours, and I am not sure if the propagation is still feasible to other cases.


3. Similar to previous points, the main concerns comes from the question, how much does the system rely on appearance instead of others like geometric cues. The objects within several experiment have a relatively clear difference in colour compared to backgrounds, which makes it a good "colour segmenter", limiting the pipeline towards complex scenes. 


4. From quantitative evaluations, the initial mask acts as an important role. Will the quality of the initial masks severely affect the final performance?  During training, will the initial segmentation loss be turned-off in the later phase of training to achieve better details?

5. As an unsupervised method, how to properly choose the loss weighting in Eq.17?

6. It would be good to demonstrate some illustrations about how each individual object-nerf works in and outside the object regions like Fig2, but for more complex scenes similar to Fig4 and Fig5.

7. Is the system performance sensitive to errors within camera poses?

8. Minor Issues.
- L157. m_k(x)=0 --> m_k(x)=1
- References should be added in L189 about the unsupervised single image-based approach, which is revealed late in the paper.
- L207. I suggest adding references or some extra explanations here. Otherwise it is hard for readers to immediately know what is exactly thee deep feature extractor here and how it works.

questions: 
-How does the system work on objects with rich textures or similar appearance to backgrounds? For example, BlenderMVS or DTU datasets? 
In addition, under this situation, will the  propagation mechanism using average colour be still helpful?

-How does the initial masks involve during training and how to balance different losses.

-Is the system sensitive to pose registration accuracy?
limitations: The authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper views image segmentation task from a 3D perspective, applies NeRF to the unsupervised multi-view image segmentation task for the first time. The paper guides the reconstruction of semantic field through the radiation field propagation, and proposes a mask refine procedure based on EM to handle complex scenes with multiple objects.
strengths_and_weaknesses: Pos:
The method is novel. NeRF is appropriatly applied to deal with multi view image segmentation tasks from the perspective of 3D, and a bidirectional photometric loss is proposed to construct the unsupervised task. The experimental improvements are remarkable.
The overall writing of the paper is smooth and the overall structure is clear.

Neg:
Lack of analysis on why propagation is useful makes it difficult to understand why it works.
When dealing with multi-object complex scenes, this method needs a strong feature extractor, which requires large-scale data for pretraining，It's hard to tell whether it's the strong feature extractor works or the paper‘s method works. Especially for the comparasion with other methods, the introduction of this large-scale external data will make the comparison unfair.
questions: The authors should explain why Propagation works intuitively?
limitations: When dealing with multi-object complex scenes, this method needs a strong feature extractor, which requires large-scale data for pretraining，It's hard to tell whether it's the strong feature extractor works or the paper‘s method works. Especially for the comparasion with other methods, the introduction of this large-scale external data will make the comparison unfair.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach to learn 3D object instance segmentation, given RGB images and camera poses. The main novelty is the "propagated" radiance field obtained by summing radiance field layers weighted by *inverted* masks. Correct labeling would maximize its photometric error. The network is trained using both traditional photometric loss and the proposed loss. EM-refinement is used to improve the result.
strengths_and_weaknesses: Strength: Unlike prior work, instance segmentation is possible. And inverting the mask is an interesting idea.

Weakness: The scenes do not seem complex enough for the task. An object close to a wall (in addition to the floor) or another object may be difficult to segment. It would be insightful to see some failure cases (either real or synthetic).
questions: Table 1 is confusing. It says “The best results without supervision are boldfaced.” But it is not clear on its own which rows they are. If I understand correctly, ablated results perform worse, as expected. And all the other papers had some label or bounding box supervision.  It might be helpful to indicate which additional supervisory signals they used, in each row.

Was this baseline considered: train vanilla NeRF, do 3D reconstruction (e.g. marching cubes), treat each connected volumetric component as an instance (assume known floor level). It would also be unsupervised.

L157, is m_k(x) = 0 supposed to be m_k(x) = 1 ?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on unsupervised segmentation in 3D with multiview images. The main idea is to use a mixture model of multiple visually contrastive radiance fields to compositionally reconstruct the scene. The contrastiveness among the RFs is enforced by propagating average appearance to empty space and maximizing the visual differences between the reconstruction and the inversely mixed composition. Experiments show good results on a single-object real dataset and a multi-object synthetic dataset.
strengths_and_weaknesses: Strength:

+ Interesting idea to create and use contrastiveness in compositional NeRFs for unsupervised segmentation.
+ In general, the experimental results look good.
+ Experiments on a real dataset.
+ Comprehensive comparison to the state of the art.

Weakness:

- I feel a few aspects in technical description and experiments are not well clarified and justified. For Eq. (4), why does it need to be differentiable? It seems l(x) is only used in Eq. (5), but is discontinuous and not differentiable. Why are numbers in 2nd last row of Table 1 missing?
questions: It would be interesting to see if the same method works with single images. Have you ever tried that with 2D or 3D RFs?
limitations: Yes
review_title: null

Meta-review: metareview: Reviewers are generally positive about the submission, and all recommend acceptance post rebuttal.  They appreciate the new formulation and the strong results.  The AC agrees and recommends acceptance. 
confidence: Less certain
Decision: Accept