Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel approach for getting a NeRF to output both (i) the standard color, and, novel for this work (ii) zero shot-trained features that can be used for semantic segmentation / decomposition / editing. Put differently, whereas previous work had parallel branches for generating, eg, closed-set semantic segmentation labels, this work generates features that can be used with standard zero-shot segmentation techniques. These features are trained via a distillation using LSeg / DINO-based teacher networks.
strengths_and_weaknesses: Positive:
- The paper is generally well written and easy to understand.
- The topic is both very relevant and novel -- there is much work on the geometric side of NeRF, but much less on semantics and even less on feature generation and semantics / editing.
- The approach is principled.

Negative:
- While the feature generation parts is fairly well explored, I found the actual editing part to not be explained very much /  very well. Unless I'm missing something, most of the text focuses on the generation of the features and the decomposition, but the editing is captured only by "We can combine this with more complex edits, including optimization-based methods like CLIPNeRF". I would very want more detail to be added here.
- While the overall structure of the proposed approach is novel (to my knowledge), the individual components are not. I would almost go as far as to say that I see this as a systems paper.

Overall I am quite positive about the paper -- (i) the methodology, while simple, is clear and appears novel; (ii) the results are very good. If accepted, I think the work might end up being quite influential.
questions: Would the approach be able to train just the feature generation branch? While that would not be very useful in the current context, it would nevertheless be interesting to know if the image branch is absolutely necessary for training the feature branch.
limitations: It would be useful to have a discussion on potential failures.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: To edit a scene more easily, the authors propose a semantic scene decomposition of NeRF by leveraging distillation techniques to distill the 2D feature extractor such as LSeg or DINO into a 3D feature field. Therefore, the distilled NeRF can segment the regions of interest with given texts.
strengths_and_weaknesses: Strengths:
1. The proposed method is easy to understand and follow.
2. The edited results on scene "Flower" looks fancy.

Weaknesses:
1. The claim is not rigorous. The authors claim "feature encoders pre-trained in a self-supervised framework on the image domain". However, this is partly true as LSeg requires semantic segmentation annotation for training.
2. As the author argues "We note that the goal of this paper is not to achieve state-of-the-art performance on 3D semantic segmentation tasks.". However, this will weaken your contribution as you already learned from a large-scale pre-trained model (LSeg), but do not plan to achieve SOTA performance even on NeRF-related datasets. Further, you compare only one 3D segmentation method in Table 1, which contradicts your claim.
3. Mixed-use of Figure and Fig.
questions: 1. Overall, the paper introduces a distillation method for NeRF semantic segmentation. Although it works well on several scenes, it is highly suggested to test your method on other indoor scenes of LLFF dataset and compared with SOTA indoor 3D semantic segmentation dataset (e.g. Scannet), as you already adopted a highly complicated 2D segmentation model.
2. The authors verify their method based on vanilla NeRF. How about the generalization ability on other NeRF variants? For example, MipNeRF, NeRF for unbounded scenes? You can try to integrate the distillation into an unbounded NeRF variant and show the segmentation quality/view consistency effect by quantitative metrics and visualized videos (e.g. on KITTI dataset with some SOTA segmentation models).
limitations: See the weakness and questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studied an important problem, i.e., editing NeRF. It argued that previous works show difficult to selectively edit specific regions or objects. Thus, the paper decomposed scene semantics by learning a semantic feature descriptor distilled from the pretrained CLIP-LSeg or DINO models. The full method DFF (Distilled Feature Field) supports simple query-based decomposition and editing.
strengths_and_weaknesses: **Strengths**
1. It is smart to exploit the distilling avenue to allow query-based decomposition and editing for NeRF.

2. The DFF framework is well presented in Figure 1 (left). And the method part clearly explains how to learn a semantic feature descriptor by taking the pretrained CLIP-LSeg or DINO models as teachers.

3. The paper examines different editing operations and shows many qualitative results to demonstrate the effectiveness of DFF.


**Weaknesses**

While the DFF approach is really interesting, the reviewer has some concerns w.r.t. the implementation details and experiments. 

1. The reviewer might miss some details, but it seems the paper has not clearly presented how to perform scene editing after training a DFF. Do we need to individually train a target scene for each editing operation? 

2. As the paper has stated that the code of complete reproduction of all the results is not yet publicly available, it would be better to include the training details in the main paper.

3. Is it possible to make some quantitate comparisons with other NeRF editing approaches?


questions: Overall, the presented method is interesting and novel. But there are some unclear points w.r.t. the implementation details and experiments. The reviewer hopes the authors can answer the questions in the "Weaknesses" part.


************************
Thanks for answering my questions. I am happy to raise my rating to "Weak Accept", and hope the authors can carefully addresses the concerns in the revision. Especially, please explain more on the implementation details and clearly present the editing details.
limitations: Possible limitations have not been discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a simple yet novel solution to achieve dense volumetric semantic segmentation on NeRF, which supports semantics-guided NeRF decomposition and local editing. This decomposition is achieved by adding an additional renderer that renders the volumetric feature field similarly to radiance by distilling from a pre-trained 2D image semantic feature encoder like CLIP-LSeg. A rich set of experiments is presented to show the decomposition quality, generalization capability to different text and even image queries, and local editing (appearance and geometry) results based on the decomposition.
strengths_and_weaknesses: ++ The major strength of this work is that it allows high-quality volumetric NeRF semantic segmentation by reusing the zero-shot image segmentation models.

++ The method is simple and easy to implement, allowing for good results on various applications and works on various NeRF scenes without needing extra training data.

++ Solid experiments and promising results, quantitatively and qualitatively.

-- The method does not show any significant technical breakthrough and is heavily based on LSeg, which limits its technical depth. But I do not think it is a major issue.
questions: I think most of the technical questions and evaluations I would like to see are addressed in the paper. Some minor questions I have:
1. The difference between image segmentation and NeRF segmentation is that image pixels are equally weighted but volumetric sample points have naturally-defined weights (density). Would this cause uncertain/noisy labels in less certain regions (empty space or inside the objects)? I know these regions might not be as important as object surfaces, but just being curious.
2. Is the algorithm robust against artifacts that are commonly seen in real-world NeRF (like misalignment and cloudy artifacts) due to less-perfect camera or input image artifacts?
3. The distillation supervision is applied on alpha-blended renderings, would this be a potential issue that the training tries to overfit the 2D feature supervision with bad volumetric features on the samples? Especially on NeRF trained on narrow baselines? The radiance may have a similar issue but the difference is that the ultimate results for radiance are still rendered views, but here we want to directly use the volumetric features.
4. Is there any boundary issues with the segmentation? Like incomplete regions or holes/noisy points?
limitations: There is not enough discussion on the limitations as far as I can find. I would suggest the authors add such discussions in the next version.
review_title: null

Meta-review: metareview: The paper proposes an approach for manipulating 3d scenes represented with implicit neural representations (NeRF-like), via distilling 2D feature extractors into a 3D feature field. The method shows convincing qualitative results on scene editing and promising quantitative results on semantic segmentation. 

All reviewers are (to a different degree) positive about the paper, noting good presentation, interesting and fairly novel approach, fairly thorough and convincing results (it would be nice to have quantitative results on scene editing, but that's quite non-trivial). 

Overall, this is an interesting and well-executed paper and I am happy to recommend acceptance.
confidence: Certain
Decision: Accept