ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
Authors: Jiahua Dong
Yu-Xiong Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: neural radiance field
diffusion model
editing
TL;DR: View-Consistency-Aware 3D Editing

Abstract: We introduce ViCA-NeRF, the *first* view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that *explicitly* propagate the editing information across different views, thus ensuring multi-view consistency. For *geometric regularization*, we leverage the depth information derived from NeRF to establish image correspondences between different views. For *learned regularization*, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is available at: https://github.com/Dongjiahua/VICA-NeRF

ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
Jiahua Dong Yu-Xiong Wang University of Illinois Urbana-Champar [jiahuad2, yxwleillinois edu
"Turn him into
Instuct-NoRFF2NoRF
ViCA-NeRF (Ours)
"Add some flowers.
Catriet-NeRFINeRR
ViCA-NeRE (Ours)
Figure Our ViCA-NeRF is the first work that achieves multi-view consistent 3D editing with text instructions, applicable across a broad range of scenes and instructions. Moreover, ViCA-NeRF exhibits controllability, allowing for early control of final results by editing key views. Notably, ViCA-NeRF is also efficient, surpassing state-of-the-art Instruct-NeRF2Nedt by being 3 times faster.
Abstract
We introduce ViCA-NeRF, the first www-coosistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, blend edits from different views to create preliminary 3D edit. This is followed by second stage NeRF training, dedicated to further refining the scene's 's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details. compared with the state of the art. Our code is available at:
edit key views and propagate
Introduction
The recent advancements in 3D reconstruction technology, exemplified by the neural radiance field (NeRF) [1] and its variants, have significantly improved the convenience of collecting real-world
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

3D data, Such progress has opened up new opportunities for the development of various 3D-based applications. By capturing RGB data from multiple views and obtaining corresponding camera parameters, NeRF enables efficient 3D representation and rendering from any viewpoint. As the availability of diverse 3D data increases, so does the demand for 3D editing capabilities.
However, performing 3D editing on NeRF is not straightforward. The implicit representation of NeRF makes it challenging to directly modify the 3D scene. On the other hand, since NeRF is trained on RGB images, it can effectively leverage a wealth of existing 2D models. Motivated by this, state-of-thhe-ar methods, such as Instr [2], apply 2D editing on individual images via iext-instructed diffusion model (e.g., Instruct-Pix Pix2Pix [3]), and then rely exclusively on NeRF learn from the updated dataset with edited images and propagate the editing information across different views. Despite yielding encouraging results, this indirect 3D editing strategy is time-consuming. because of its iterative process that requires cyclically updating the image dataset while progressively fine-tuning NeRF parameters. And more seriously, due to the lack of inherent 3DD structure, the editing results from the 2D diffusion model often exhibit significant variations across different views. This raises critical concern regarding 3D inconsistency when employing such strategy, particularly in cases where the editing operation is aggressive, e.g., "Add some flowers" and "Turn him into iron man" " as illustrated in Figure 1.
To overcome these limitations, in this work we introduce ViCA-NeRF, a View-Consistencyy NeRF editing method. Our key insight is to exploit two additional sources of regularization that explicitly connect the editing status across different views and correspondingly propagate the editing information from edited to unedited views, thus ensuring their multi-view consistency. The first one is geometric regularization. Given that depth information can be obtained from NeRF for free, we leverage it as guidance to establish image correspondences between different views and then project edited pixels in the edited image to the corresponding pixels in other views, thereby enhancing consistency. This approach allows us to complete preliminary data modifications before fine-tuning NeRF, as well as enabling users to determine 3D content by selecting the editing results from key 2D views. Moreover, our method allows for direct and consistent edits, it is more efficient by avoiding the need for iterative use of the diffusion model.
The second one is learned regularization. The depth information derived from NeRF can often be noisy, yielding certain incorrect correspondences. To this end, we further align the latent codes in the 2D diffusion model (e.g., Instruct- Pix2Pxx between edited and unedited images via blending refinement model. Doing so updates the dataset with more view-consistent and homogeneous images, thereby facilitating NeRF optimization.
Our ViCA-NeRF framework then operates in two stages, with these two crucial regularization strategies integrated in the first stage. Here, we edit key views and blend such information into every viewpoint. To achieve this, we employ the diffusion model for blending after a simple projection mixup. Afterward, the edited images can be used directly for training. An efficient warm-up strategy also proposed to adjust the editing scale. In the second stage, which the NeRF training phase, we further adjust the dataset. Our method is evaluated on various scenes and text prompts, ranging from faces to outdoor scenes. We also ablate different components in our method and compare them with previous work [2]. Experimental results show that ViCA-NeRF achieves higher levels of consistency and detail, compared with the state of
the art.
Our contributions are three-fold. (1) We propose ViCA-NeRF, the first work that explicitly enforces multi-view consistency in 3D editing tasks. (2) We introduce geometric and learned regularization, making 3D editing more flexible and controllable by editing key views. (3) Our method significantly improves the efficiency of 3D editing, achieving a speed 3 times faster than the state of the art.
2 Related Work
Text-to-Image Diffusion Models for 2D Editing. Diffusion models have recently demonstrated significant capabilities in text-to-image editing tasks [3-8]. Earlier studies have leveraged pre-trained text-to-image diffusion models for image editing. For instance, SDEdit [8] introduces noise into an input image and subsequently denoises it through a diffusion model. Despite achieving reasonable results, this approach tends to lose some of the original image information. Other studies have concentrated on local inpainting using provided masks [4, 5], enabling the generation of corresponding content within the masked area with text guidance. More recently, Instruct-Pix22 [3] proposes an
2

---

Stagel: Dataset Editing
Text: Turn him into
Iron Man
Stage2: NeRF Training
Instruct-Fix2Pxx
Key Views
Mixup
Blonding
Depth
Source
Original Images
Overview of our ViCA-NeRF. Our proposed method decouples NeRF editing into two stages. In the first stage, we sample several key views and edit them through Instruct- Pix2Pxx Then, we use the depth map and camera poses to project edited keyframes to other views and obtain mixup dataset. These images are further refined through our blending model. the second stage, the edited dataset is directly used to train the NeRF model. Optionally, we can conduct refinement to the dataset according to the updated NeRF
Figure 2:
approach editing images based text instructions. Dustruct-Pix2P1 accepts original images and text prompts as conditions. Trained on datasets of editing instructions and image descriptions, this approach can effectively capture consistent information from the original image while adhering to detailed instructions.
In this paper, adopt Instruct-Pix2Pix
editing model. Importantly, further explore the potential of Pestruct-Pix2Pix by extending its application to editing, blending, and refinement tasks in more challenging 3D editing scenarios.
Implicit 3D Representation. Recently, neural radiance field (NeRF) and variants have demon- strated significant potential in the field of 3D modeling [1. 9-14). Unlike explicit 3D representations such as point clouds or voxels, NeRF uses an implicit representation to capture highly detailed scene geometry and appearance. While training conventional NeRF model can be time-consuming, Instant- NGP [10] accelerates this process by training a multi-layer perceptron (MLP) with multi-resolution hash input encoding. Notably, NeRFStudio [9] offers comprehensive framework that handle various types data. It modularizes each component of NeRF, thereby supporting a broad range of data. These efforts bridge the gap between image data and 3D representation, making it possible to guide 3D generation from 2D models. Our work uses the 'nerfacto' model from NeRFStudio to fit real scenes. We also leverage the depth estimated from the model maintain consistency.
3D Generation. Inspired by the success of 2D diffusion models [15, 16), 3D generation has drawn great attention in recent years [17-21]. Dream Fusion first proposes to
score distillation sampling (SDS) loss as the training target. Later, Magic3D [20] generates higher resolution results by coarse-to-fine procedure. Various scenarios have been explored, including generating detailed mesh from text prompts [21] and generating novel objects based on images from a few views [18].
Rather than generating new content, our work focuses on editing the 3D representation with text prompts. By explicitly leveraging 2D editing, we provide more direct approach to control the final 3D result.
NeRF Editing. Because of the implicit representation of NeRF [1], editing NeRF remains challenge. Early attempts [22-32] perform simple editing on shape and color, by using the guidance from segmentation masks, text prompts, etc. NeuralEditor [33] proposes to exploit point cloud for editing and obtain the corresponding NeRF representation. Another line of work tries to modify the content through text prompts directly. NeRF-Art [34] uses pre-trained CLIP [35] model serve as an additional stylized target for the NeRF model. More recently.
[2] and Instreut- -D-ttt--t [36] leverage instruct-Pix2Pa [3] for editing. They can edit the content of NeRF with language instructions. However, both require using the diffusion model iteratively in the NeRF training process, making them less efficient. More importantly, though it shows promising results on real-world data, consistent results in 3D. Thus, unable to generate a detailed result where 2D edits tend to be different, and it also suffers from making diverse results from the same text prompt.
can only work when the 2D diffusion model produces

---

Image Mixup
Blending Model
Edttd
Text: Turn into Iron
Cond
Text Cond
Mixup Image
Pullusion Diffusion
Moan
Moan
D
Ditupion
Projaction Mixup
Blended Image
Image Cond
Image Cond
Origin
Depth
Modilee Imp2o avarage
Original Images
Mixup Image
Figure 3: Illustration mixup procedure and blending model. We first mix up the image with the edited key views. Then, we introduce blending model to further refine it. The blending model utilizes two modified Instruct-Pix2Pix ('Inp2p') processes. In each process, we generate multiple results and take their average on the latent code decode single final result.
In our work, we propose to update the dataset at the start of training and further refine it once, making it significantly more efficient. In addition, Dnstruct-NeRF2NeR As several key views guide the whole 3D scene, we allow users to personalize the final effect NeRF by choosing 2D edit results.
depth guidance for detailed 3D editing. Our ViCA-NeRF only needs
method results in more detailed and diverse edit results compared with
3 Method
Our ViCA-NeRF framework is shown in Figure 2. The main idea is to use key views to guide the editing of other views, ensuring better consistency and detail. Given the source NeRF, first extract each view's 's depth from and sample a set of key views. Then, we edit these key views using Dnstruct-Pix2Pin [3] and propagate the edit result to other views. To address the problem of incorrect depth estimation from NeRF, we perform mixup on each image and then refine it through blending model. The obtained data can be used directly for NeRF training. Optionally, we can perform post-refinement to the dataset after obtaining more consistent rendering results from NeRF.
3.1 Preliminary
We first elaborate on NeRF, including how we obtain rendered RGB images and coarse depth from the NeRF model. Subsequently, present Instruct-NeRF2NNRREF diffusion model for 21) editing. With this model, we can perform 2D editing using text and then further enable 3D editing.
Neural Radiance Field. NeRF [1] and its variants [9, 10] proposed for 3D scene representation. The approach models a 3D scene as 5D continuous function, mapping spatial coordinate (x, y, and viewing direction (0, Φ) to volume density and emitted radiance For given set of 2D images, NeRF is trained to reproduce the observed images. This achieved by volume rendering, where rays traced through the scene and colors accumulated along each ray to form the final rendered image. While NeRF does not explicitly predict depth, one common approach is to compute weighted average of the distance values along each ray, where the weights are the predicted volume densities. This process can be conceptualized as approximating the depth along the ray.
Instruct-Pix2Pis. Instruct-Pix2P1t [3] is 2D diffusion model for image editing. It takes text prompt eT and a guidance image e as conditions. Given an image zo, it first adds noise to the image to create 2t, where t is the timestep to start denoising. With denoising U-Net eg. the predicted noise can be calculated as
ē= t7,
When predicted noise is calculated, Dnstruct-Pix2Pix follows the standard denoising process to update the image 2, step by step and obtain the final result 2p, which is an edited image consistent with the initial image and text instruction.

---

"Add some Blowerss
"Make looks lhe snoweed
Spider
Cyberpnnt Style
"Bur bear into grinty Bear"
Original NeRF
Instruct-NeRF2MERF
Instruct-NeRF2NERF
Ours
Figure 4: Qualitative comparison with Instruct-NeRF2NeRF: Our ViCA-NeRF provides more details compared with Instruct-NeRFNNeRFF In addition, our ViCA-NeRI handle challenging prompts, such as "Turn him into robot," whereas Instruct-NeRF2NeRR fails in such cases.
3.2 Depth-Guided Sequential Editing
our approach, we choose to reflect the user's edits in 3D. Thus, we propose sequential editing strategy. By defining certain key views, we sequentially edit them by projecting the previous edits to the next key view. This is the geometric regularization that explicitly enforces the consistency between different views.
Sequential Editing with Mixup. The overall procedure of editing is simple yet effective, which consists of editing key views and projecting to other views. First, we select a sequence of key views starting from random view. Then, we sequentially use Instruct-Pix2P2 to edit each view and project the modification to the following views. At the same time, non-key views are also modified by projection. After editing all key views, the entire dataset is edited.
Given that we project several key views onto other views when they are edited, each view could be modified multiple times. Only pixels that have not yet been modified are altered prevent inadvertent overwriting. Therefore, modifications from key views are mixed with each view.
Consequently, method yields final result that aligns closely with the modified views, facilitating a controllable editing process. Since previously modified pixels are not changed until blending refinement (explained in Section 3.3), the edits of the first few views dominate the final result.
employ key views make the editing process consistent. This strategy allows the entire scene to be constrained by handful of edited views. To ensure the edited views maintain consistency and bear similar content, we define the modified ratio p for each view as the fraction of modified pixels. Then, we use the following equation to select the view with the largest
Key View Selection.

---

weight w as the next key view:
( --)
P p 0
where @ signifies the maximum desired ratio, which is set to Φ 0.3 in all experiments. Intuitively. each time the next selected key view will have a suitable overlap with the edited region from each of the previous key views. The key view selection ends when all views' minimum p exceeds a certain threshold.
Deidh- Projection. Modern NeRF models are capable of generating depth maps directly, thereby making it possible to construct geometric relationships between different views. We illustrate an example of projecting from edited view view i as below.
Given an RGB image and its corresponding estimated depth map D,, the 3D point cloud associated with view can be calculated as:
Pj = PD; K.)
Here. K; denotes the intrinsic parameters, & represents the extrinsic parameters, and p-1 is function mapping 2D coordinates to 3D point cloud.
To find the corresponding pixels another view, project this point cloud to view j as follows:
4 K,,
(4)
where 4t signifies the color of the pixels projected from 1y onto 4. In practice, to ensure the correct correspondence and mitigate the effects occlusion, we compute the reprojection error to filter out invalid pairs. By using this correspondence, can project pixels from view to view i.
3.3 Blending Refinement Model
The depth information derived from NeRF [1] can often be quite noisy, resulting in noticeable outliers, so only applying the geometric regularization is insufficient. Specifically, while we can filter out incorrect correspondences, the remaining pixels and projections from varying lighting conditions may still introduce artifacts. Furthermore, a reasonable 2D edit in a single view may appear strange when considered within the overall 3D shape. To address these challenges, we introduce a blending refinement model that employs two Instruct-Pix2PP [3] processes, acting the learned regularization to improve the overall quality.
Details about the model are shown in Figure 3, where each diffusion process serves unique functions. During the first pass, we use the mixup image input and the original image as a condition. This procedure aims to purify the noisy mixup image by preserving the structural integrity of the original image. The resulting image appears dissimilar to the consistent mixup image but without any noise. Thus during the second pass, we use the resulting image from the first pass as input and the mixup image as the condition. Given the shared semantie information between these two images, the outcome strives to closely align with the mixup image's detailed characteristics. Doing so allows us to swiftly generate a clean, stylized result, bypassing the need for iterative updates the dataset during training.
An essential component of our refinement model involves using an average latent code for each of the two processes. We discovered that the diffusion model tends to generate diverse results, making challenging to closely align with the target structure or content. However, this effect can be substantially mitigated by averaging multiple runs. Specifically, we introduce and update them independently. Finally, we compute the average of the latent codes and decode back into the image. This method provides a more stable and consistent output, enhancing the overall quality of the editing process.
noised latent codes
3.4 Warm-Up and Post- Rest-Refinement Strategies
To make the editing more efficient and further improve the quality, we propose a warm-up strategy along with post-refinement strategy.
Warm-Up. When editing via instruct-Pix2Pic the extent of editing changes can be modulated through hyperparameters, such as the guidance scale [15]. However, achieving the desired overall

---

NatP-4n
VICA-NaR (har Intras--GPPNN
Original
Cligh
Mdral packicg
Figure 5: Comparison on NeRF-Art. We compare the editing results based on NeRF-Art's [34] sequences and edits. Our ViCA-NeRF produces more detailed information and achieves more substantial changes to the content.
style remains problem. Although Instruct-NeRF2Nel ensures consistent stylization, it necessitates iterative updates, which could inefficiently involve a large number of iterations.
To overcome these limitations, introduce strategy efficiently warm up the editing process by blending edits directly. This approach is performed before the entire editing process. Specifically, we randomly select a view edit, and then project the edited view onto all other views. For given view i and an edited view j, an update is calculated as follows:
1 = AI, @ m. + (1 - A)4., O mw + 14 O mw).
(5)
Here, my denotes the binary mask for the projected area, and is a hyperparameter employed to control the ratio preserved from the original pixel value. This efficient warm-up strategy accelerates the editing process while preserving high degree of consistency in the stylization.
Rost-Refinemen.. To further enhance the consistency of the edited results, we employ post- refinement strategy once the NeRF has been trained on our updated data. We still utilize the modified Instruct-Pix Pix2Pix with averaging architecture to generate a more accurate target for each view. Unlike the first stage, this time we employ the rendered image as input while continuing to use the mixup image condition. This post-refinement strategy further refines the consistency and quality of the final output, contributing to more cohesive and visually appealing result.
Experiments
We conduct experiments on various scenes and text prompts. All our experiments are based on real scenes with NeRFStudio [9]. We first show some qualitative results and comparisons between method and NoRF [2]. For artistic stylization, we test our method with the cases from NeRF-Art [34]. Experiments show that we achieve more detailed edits. Based these scenes, further conduct ablation studies on our method, including the effects of different components the framework, the warm-up strategy, failure cases, and representative hyperparameters. We also conduct quantitative evaluation of the results, testing the textual alignment, consistency, and efficiency of our method.
4.1 Implementation Details
For fair comparison, our method employs configuration similar to that of Instruct-NeRF2NNee We utilize the 'nerfacto' model from NeRFStudio and Aestruct-ixxPii, [3] as our 2D editing model. For the diffusion model, we apply distinct hyperparameters for different components. During the editing of key views, the input timestep set within the range [0.5, 0.9]. We employ diffusion steps for this phase. In the blending refinement model, we set to 0.6 and n,r to 5, and use only 3 diffusion steps. The reduced number of steps enables the model to operate more quickly, since encoding and decoding are executed only once. The diffusion model provides additional adjustable
7

---

Editid 2D Key View
Rendered Result
Edited 2D Key View
"Torn him into Link from Zolda"
Figure 6: Different 2D edits and corresponding 3D edits. Our ViCA-NeRF achieves high corre- spondences between 2D edits and 3D edits, thus enabling users to control the final edits. On the other hand, our method exhibits high diversity with different random seeds.
Origin Image
Mixup
Blended
Post- Post-Refined
Rendered
Improving Consistency
Figure 7: Image update through our ViCA-NeRF. Our blending model effectively eliminates artifacts and boundaries caused by mixup. The post-refinement step further improves the consistency with the mixup image and the 3D content.
hyperparameters, such as the image guidance scale S7 and the text guidance scale Sy. We adopt the default values of S, 1.5 and ST 7.5 from the model without manual adjustment.
For training the NeRF, we throughout the process. Initially, we pre-train the model for 30,000 iterations following the 'nerfacto' configuration. Subsequently, we continue train the model using our proposed method. The optional post-refinement process occurs at 35,000 iterations. We obtain the final results at 40,000 iterations.
L1 and LPIPS (Learned Perceptual Image Patch Similarity) loss
4.2 Qualitative Evaluation
Our qualitative results, illustrated in Figure 4 and Figure 5, demonstrate the effectiveness of our method in editing scenes through multiple views. We present results on three scenes with varying complexity, from human figures to large outdoor scenes. We also evaluate the stylization capabilities of NeRF-Art [34] and compare our results with the previous state of the art.
For comparison with Instruct-NeRF2NeRR we replicate some cases from their work and assess For simpler tasks such "turn the bear into grizzly bear" or "make
several more challenging it look like it has snowed," our method achieves results similar to those of Instruct-NeRF2NeRRF This may be due the inherent consistency of diffusion models in handling such scenarios. For more complex tasks that require additional detail, like transforming an object into a robot or adding flowers, our method shows significant improvements. A notable example is the "Spiderman cas," where Instruct-NeRF2Ner struggles with differentiating lines on the mask, while our method renders clear appearance.
When compared on NeRF-Art, our method clearly outperforms the previous techniques by exhibiting more details and adhering closely to the desired style. Although model as Dnstruct-NeRF2NeeR we are able to produce more vivid colors and details, which are not achievable with iterative updates.
the same 2D editing
Moreover, significant advantage of our method that the final edits can directed through several key views. This allows us to achieve consistent edits not only across different 3D views but also between the 2D edits and the 3D renders, as shown in Figure
4.3 Ablation Study
In our ablation study, we first evaluate the individual effects mixup. blending, and post-refinement on the quality of editing. Further investigation includes the impact of warm-up iterations and NeRF

---

Origin Image
warmup=0
warmu=100
Figure 8: Edit distribution with different warm-up configurations. When we use warm-up, the edit distribution becomes closer to the target distribution.
depth estimation accuracy. We also explore the influence of the parameter & on key view selection for efficient editing. Each of these components is crucial for understanding the nuances of our method's performance.
Model Component Ablation. As shown in Figure 7, the mixup image may initially appear very noisy and contain artifacts. When we apply blending model to refine it, the boundary from copy-pasting and artifacts is removed. However, while maintaining almost all the original content, the appearance of the image still changes slightly. Finally, when apply our post-refinement strategy update it, the result improves and becomes similar to the rendered result. Notice that the result after blending sufficiently good, making further refinement an optional step.
Warm-Up Iteration Ablation. Considering that Instruct-Pix2Pix relies heavily on the input image, modifications to the input significantly affect the output distribution. Therefore, to validate warm-up up procedure, which is designed to alter the scale of editing, we adjust this parameter from (no warm-up) to 30. As Figure 8 illustrates, our findings indicate that while preserving the primary geometry, our warm-up procedure can effectively act as controller for adjusting the scale of editing, introducing only minor computational costs.
The projection procedure is critical in our framework; therefore, inferior depth estimation may affect the final result. We conduct ablation study focusing on two prevalent scenarios: glossy surfaces and sparser views. As Figure 9a demonstrates, our model can accurately make modifications to the bookshelf. If the reprojection check is omitted, more extensive changes are possible, albeit at the cost of losing original details. With sparser views, as depicted in Figure 9b, model is capable of editing provided that the NeRF is successfully trained.
Ablation for Depth Estimation Errors.
Ablation on Key Views. We perform experiments based on the hyperparameter ø. Specifically, we test the results with é set to O, 0.3, and 1, as illustrated in Figure 10. When ø = 0, the view with the least overlap is selected, resulting clear boundary (highlighted in the red box). In other cases, this issue is mitigated. However, when & 1, nearly all views are designated as key views, which leads to reduced efficiency.
4.4 Local Editing
Existing methods often induce simultaneous changes in both background and foreground during editing, which contradicts specific user intentions that typically focus on altering distinct objects.
sheal
VIC. MAR.
20% (1.3) views
50% (33) views
50% (59) views
"Rum face in stul"
(a) Results with glossy surface
(b) Ablation study input views
Figure 9: Ablation study with incorrect depth. We study both glossy surfaces and sparser views. where the results demonstrate that as long as NeRF is built successfully, our model can manage incorrect depth estimation.

---

$ 0.3 "Rurn him into spider man"" Figure 10: Ablation study on ø in keyframe selection. Our keyframe selection successfully avoids the boundary problem and achieves efficiency.
propogata
trained NeRF
W/O local edit "Tom his face in to shual
wi local edit
Local mask propagation
(b) Local editing
Figure 1I: Application for local editing. Our ViCA-NeRF can be applied with local editing. The edits can be done for the targeted object without changing the background (e.g., wall).
To address this discrepancy, we introduce a straightforward, user-friendly local editing technique within our framework. Initially, we employ Segment Anything Thing (SAM) [37] to derive masks encompassing all components within each image. Subsequently, users pinpoint the target instance with a single point to isolate it. Following this, we utilize the depth information to project the identified mask across different views progressively. This process continues long as the projected mask sufficiently overlaps with the corresponding mask in the current view. Then, we integrate the wrapped segmentation mask with the segmented masks the current view.
Figure 11 exemplifies the process, resulting in a multi-view consistent segmentation that permits exclusive modification of the human figure, leaving the background intact.
4.5 Discussion
For quantitative evaluation, we adopt the directional score metric utilized Instruct-Pix2PP [3] and the temporal consistency loss employed in text prompts from Instruct-NeRF2NeRk we establish various difficulty levels to assess performance. Results and comprehensive details provided in the supplementary material.
[2]. Due the undiselosed split and
further benefit of our approach the decoupling of the NeRF training from the dataset update process. This separation significantly reduces the computational burden typically imposed by the diffusion model at each iteration. The only computational expense arises from warm-up, blending, and post-refinement stages. We analyze the time cost of the "Face" scene and compare it with NnsF2NE on an RTX 4090 GPU, Both methods undergo 10,000 iterations; however, Instruct-NeRF2NeRE requires approximately 45 minutes, whereas our ViCA-NeRF only needs 15 minutes.
Limitations. Our method's efficacy is contingent upon the depth map accuracy derived from NeRF, which underscores our reliance on the quality of NeRF-generated results. Although our blending refinement mitigate the impact of incorrect correspondences to some extent, the quality of the results deteriorates with inaccurate depth information. also note that akin to Instruct-NeRF2Ng the edited outcomes tend to exhibit increased blurriness relative to the original NeRF. We investigate this phenomenon further and provide a detailed analysis the supplementary material.
Conclusion
In this paper, we have proposed ViCA-NeRF, a fiw-coonsstency--wwar 3D editing framework for text-guided NeRF editing. Given text instruction, we can edit the NeRF with high efficiency, In addition simple tasks like human stylization and weather changes, we support context-related operations such as "add some flowers" and editing highly detailed textures. Our method outperforms several baselines on a wide spectrum of scenes and text prompts. In the future, we will continue to improve the controllability and realism of 3D editing.
10

---

Acknowledgements
This work was supported in part by NSF Grant #2106825, NIFA Award #2020-67021-32799, the Illinois-Insper Partnership, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, the IBM-IIlinois Discovery Ac- celerator Institute, and the Amazon Research Award. This work used NVIDIA GPUs NCSA Delta through allocations CIS220014 and CIS230012 12 from the Advanced Cyberinfrastruccure Coordination Ecosystem: Services & Support (ACCESS) program, which supported by NSF Grants #2] 38259, #2138286, #2138307, #2137603, and #2138296.
References
[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021. 1, 4,
[2] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-NeRF2NeRF: Editing 3D scenes with instructions. In ICCV, 2023. 2, 3, 7, 10
[3] Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023. 2, 3, 4, 6, 7, 10
[4] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[5] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022. 2
[6] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
[7] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompi image editing with cross attention control. In ICLR, 2023.
[8] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 2
[9] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In
SIGGRAPH, 2023. 3, 3,
[10] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 11:4):102:1-1022 2022. 3.4
[11] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2020.
[12] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[13] Ricardo Martin Bruual, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[14] Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Eloo Xiaowei Zhou, and Noah Snavely. Neural 3D reconstruction in the wild. In SIGGRAPH, 2022. 3
[15] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 6,6
[16] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 3
[17] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation. In ICCV, 2023. 3

---

[18] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Ben Mildenhall, Nataniel Ruiz. Shiran Zada, Kfir Aberman, Michael Rubenstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. Dreambooth3ID Subject-driven text-to-3D) generation. ICCV, 2023. 3
[19] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using
2D diffusion. In ICLR, 2023.
[20] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang. Karsten Kreis, Sanja Fidler. Ming- Yu Liu, and Tsung- Yi Lin. Magic3D: High-resolution text-to-3D content creation. In CVPR, 2023.
[21] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Tex12Mesh: Text-driven neural stylization for meshes. In CVPR, 2022. 3
[22] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In ICCV, 2021. 3
[23] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. In NeurlPS, 2022.
[24] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In ICCV, 2021.
[25] Pei-Ze Chiang, Meng-Shiun Tsai, Hung-Yo Yu Tseng, Wei-Sheng Lai, and Wei-Chen Chiu. Styliz- ing 3D scene via implicit representation and hypernetwork. In WACV, 2022.
[26] Hsin-Ping Huang, Hung- Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan Yang. Learning to stylize novel views. In ICCV, 2021.
[27] Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, and Lin Gao. StylizedNeRF: Consistent 3D scene stylization as stylized NeRF via 2D-3D mutual learning. In CVPR, 2022.
[28] Qiling Wu, Jianchao Tan, and Kun Xu. PaletteNeRI: Palette-based color editing for NeRFs, In CVPR, 2023.
[29] Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang. and Zhaopeng Cui. SINE: Semantic-drie driven image-bases based NeRF editing with prior guiidd editing field. In CVPR, 2023.
[30] Kai Zhang, Nick Kolkin. Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. ARF: Artistic radiance fields. In ECCV, 2022.
[31] Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. SNeRF: Stylized neural implicit representations for 3D scenes. In WACV, 2022.
[32] Clément Jambon, Bernhard Kerbl, Georgios Kopanas, Stavros Diolatzis, Thomas Leimkühler, and George Drettakis. NeRFshop: Interactive editing of neural radiance fields. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(1), 2023.
[33]
Jun-Kun Chen, Jipeng Lyu, and Yu-Xiong Wang. NeuralEditor: Editing neural radiance fields
via manipulating point clouds. In CVPR, 2023. 3
[34] Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics, pages 1-15, 2023. 3, 8
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[36] Hiromichi Kamata, Yuiko Sakuma, Akio Hayakawa, Masato Ishii, and Takuya Narihira. Instruct 3D-to-3D: Text instruction guided 3D-to-3D conversion. arXiv preprint 15330
2023. 3
[37] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Ren-Yen/ Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023. 10
12


Reviews:
[{"review_id": "WMaRDlvtTF", "reviewer": "Reviewer_x1ev", "summary": "The paper proposes a method for 3D editing via text instruction that is view consistent. It leverages on the idea from Instruct-NeRF2NeRF, but additionally propagates the edits into multiple views in order to achieve multi-view consistency. To achieve such, a geometry regularization and a learned regularization were used, where the former uses depth for correspondence matching and the latter uses the typical 2D prior as used by Instruct-Pix2Pix but instead uses the average of the latent codes. Experiments show the effectiveness of the proposed method on 10 scenes.", "questions": "A critical question I had is the details of the 10 training scenes, specifically, how many input views are used for training? The number of input views would give an idea on how critical the edit propagation to the different views is. Moreover, both the edit propagation and the geometric regularizer rely on depth inferred from the NeRF, and the correctness of NeRF depth is highly dependent on the number of training views. The failure case of the original Instruct-NeRF2NeRF is when the 2D diffusion samples is not consistent in 3D, which is also dependent on how many views are there. Perhaps an ablation on this would also be meaningful and make the contribution more convincing.", "limitations": "The authors included a limitation section in the supplementary material.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a way to enable 3D editing through text instruction that is multi-view consistent. The motivation of the proposed pipeline is sound and makes sense. It is also simple, which makes it easy to follow. Experiment demos show nice qualitative examples as well as qualitative ablations.", "weaknesses": "The main concern I have with the paper is the thoroughness of the experiments. Specifically, in the main paper only qualitative results are included, and the scenes and examples presented are limited. The supplementary included a quantitative metric, but this is also only on 10 scenes.  It also mentioned that the pipeline is heavily reliant on depth, which makes one wonder whether the limited scenes shown is because of the failure cases with depth. Including failure cases might be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank the reviewer for the valuable feedback. Below we address all the concerns.\n\n> Q1. Thoroughness of the experiments; the scenes and examples presented are limited\n\nA1. While we acknowledge the reviewer’s concern that the evaluated scenes might not be as numerous as one would expect, we would like to kindly point out that we have **evaluated more scenes and examples when compared with previous works**, such as Instruct-NeRF2NeRF [Ref1]. In fact, the lack of a large-scale benchmark is a common issue in NeRF-based 3D editing research, which is not specific to our paper. Constructing a benchmark with a large number of scenes is an important yet non-trivial work for future research, which is beyond the scope of this paper.\n\nWhile our focus is not on creating a large-scale benchmark, we did perform an experimental evaluation as thorough and comprehensive as possible. Specifically, (1) in addition to assessing all the scenes and examples presented in the SOTA Instruct-NeRF2NeRF, we extended our evaluation in the submission to include additional scenes such as the “person in the farm” scenario. (2) We validated our method **across a wide range of scenes**, from object-centric (e.g., “face”) scenes to in-the-wild unbounded (e.g., “campsite”) scenes. As noted by Reviewer CQtY, \"the proposed method is general, being applicable to humans, objects, and even unbounded scenes.” (3) In **Fig. R7** of the rebuttal PDF, we have further expanded our evaluation to include extra scenes and examples. We adopt the existing NeRFStudio [Ref2] data and also transfer the data from NeRFRen [Ref3] with COLMAP. Again, this evaluation underscores that our method can be directly applied to various scenes and consistently achieve superior 3D editing results. \n\nMoreover, we would like to reiterate that we conducted both qualitative and quantitative evaluation in the submission. Due to the nature of the editing tasks, the ground truth for editing results is typically unavailable, partially making it difficult to conduct large-scale quantitative evaluation. Therefore, prior works like Instruct-NeRF2NeRF primarily focus on qualitative comparisons. As noted by Reviewer EWGc: “This is not a problem of this paper, it’s generally an issue with editing papers where there is no \"correct\" solution.” To further show the superiority of our method, we conduct a user study that evaluates both the diversity of editing results and how well they match the provided text description. We include the comparison with Instruct-NeRF2NeRF in **Fig. R4** of the rebuttal PDF. Our method achieves significantly higher preference in the study.\n\n> Q2. The pipeline is heavily reliant on depth and failure cases\n\nA2. First, we would like to clarify that while our method utilizes depth information inferred from NeRF, this does not indicate a significant dependence on the depth quality. Instead, our method is designed to be robust and not prone to failure when dealing with inaccurately estimated depth. This robustness is achieved through our blending model, which effectively removes the artifacts and guides the editing process towards a more consistent direction. Our empirical evaluation in Fig. 7 validates that our blending model successfully fixes the artifacts caused by incorrect depth and obtains reasonable geometry. \n\nWe further investigate the robustness of our method against inaccuracy or failure in depth estimation in **Figs. R3 and R5** of the rebuttal PDF. We consider 2 types of cases: glossy surface and sparse view inputs. The glossy surface, e.g. glass, will cause incorrect depth. For sparse views, as the information of view consistency is reduced, the depth will become less accurate. **Notably, while the accuracy of depth will affect the final performance, we can still successfully edit the scenes as long as the NeRF is successfully trained**. \nMoreover, we would like to clarify that, as explained in A1, the scenes evaluated in this paper were not selected with the intention of avoiding depth-related failure cases. Rather, these scenes were assessed in accordance with prior works and to make a fair comparison. \n\n> Q3. Number of input views\n\nA3. Thank you for your suggestion. We are following the same settings as Instruct-NeRF2NeRF for a fair comparison. 90% of the images are used for training and 10% are used for validation. The number of input views varies, depending on the scenes. For small scenes, e.g. “face”, it contains 59 input views. For larger scenes, e.g., “person in the farm”, it contains 343 input views. \n\nPer the reviewer’s request, in **Fig. R5** of the rebuttal PDF, we have included an ablation study to investigate the impact of using a smaller number of input views, as this can result in worse depth estimation results. As we are using nerfacto [Ref2] which is not designed for dealing with sparse views, the training of NeRF will be broken when only 20% images are used (see Fig. R5). However, as long as the NeRF is successfully trained, even with less input views, our approach is still effective. By leveraging a more powerful base NeRF model that is designed to be able to tackle sparse views like Sparf [Ref4], our method is expected to work reliably with a small number of views.\n\n[Ref1] Ayaan Haque, et al. Instruct-NeRF2NeRF: Editing 3D scenes with instructions. arXiv, 2023.\n\n[Ref2] Matthew Tancik, et al. NeRFStudio: A modular framework for neural radiance field development. SIGGRAPH, 2023.\n\n[Ref3] Yuan-Chen Guo, et al. NeRFRen: Neural radiance fields with reflections. CVPR, 2022.\n\n[Ref4] Prune Truong, et al. Sparf: Neural radiance fields from sparse and noisy poses. CVPR, 2023.\n\n"}, {"review_id": "TFQf5YkSc6", "reviewer": "Reviewer_EWGc", "summary": "This paper presents a method for editing Neural Radiance Fields using text instructions. Starting from an initial trained NeRF, the method samples a set of key views which are then edited using Instruct-Pix2Pix. Given the depth maps rendered from the NeRF, the method computes correspondences between different views and propagates the edits to the remaining views. To minimize the artifacts from warping the paper proposes to use a blending procedure that involves 2 Instruct-Pix2Pix models.", "questions": "1. It is very hard to tell if the method is significantly better than Instruct-Nerf2Nerf. The quantitative evaluation in the Supplementary shows very similar numbers. The text-image direction score and the temporal consistency score are correlated with the perceptually better results but there is no direct relationship, so it's hard to quantify which method is better. At the same time, it's hard to tell from a handful of qualitative results which method is better. This is not a problem of this paper, it's generally an issue with editing papers where there is no \"correct\" solution. I do not have a good solution for this. \n2. The keyframe selection criterion seems a little bit arbitrary. How does the value of $\\phi$ affect the final result?", "limitations": "The authors have an extended discussion on the limitations of the method which I appreciate. There is also a discussion on the societal impact in the supplementary material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies a known issue with performing per-frame edits of Neural Radiance Fields (inconsistencies between views) and proposes an intuitive method to address them. This involves selecting key frames and then use the NeRF geometry to propagate changes to the remaining views. The approach overall is technically sound.\n2. Both the qualitative and quantitative results are good, and are on-par with Instruct-Nerf2Nerf. Given that the proposed method is about 3 times faster than Instruct-Nerf2Nerf then it is a valid, more lightweight alternative.\n3. The paper is well-written and easy to follow. The figures are clear and most components of the method are adequately explained in the text.", "weaknesses": "1. The paper fails to show that all components of the method are needed. For example by looking at Figure 6, I cannot seem to see any difference between the \"blended\" and \"post-refined\" images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Below we address all the concerns.\n\n> Q1. Cannot seem to see any difference between the \"blended\" and \"post-refined\" images in Fig. 6\n\nA1. The post-refinement process is designed to further improve consistency. While it may not make a significant difference, post-refinement can improve the consistency of 2D edits for the dataset. For example, in Fig. 7 (we assume the reviewer is referring to Fig. 7 instead of Fig. 6, as Fig. 7 provides the ablation study), after post-refinement,  the man’s hat becomes more consistent with the rendered result, displaying a shape and size that align mode closely with the rendered result. In Figure IV of the supplementary, after post-refinement, the reactor on Iron Man's suit exhibits enhanced and clearer details than before, with more distinct boundaries.\n\n> Q2. Hard to tell quantitatively if the method is significantly better than Instruct-Nerf2Nerf\n\nA2. We greatly appreciate the reviewer’s comment on the common evaluation issues in the field of NeRF-based 3D editing. While we followed the prior work on using the text-image direction score and the temporal consistency score as quantitative metrics, we agree with the reviewer that they are correlated with the perceptually better results but there is no direct relationship.\n\nTo better quantify which method is better and clarify the reviewer’s concern, we further conducted a user study on the editing results and compared with Instruct-NeRF2NeRF, regarding the diversity of the editing results and how well they match the text description. The user study quantifies the comparison while being directly relevant to the perception. More detail is provided in the rebuttal PDF. And the result is shown in **Fig. R3** of the rebuttal PDF. **Our method achieves significantly higher preference in the study**. \n\nMoreover, we would like to emphasize that our main focus is to improve diversity, controllability, and consistency of 3D editing. Specifically, (1) as shown in Figs. I and II of the supplementary, as well as other qualitative results, we are able to achieve edits where Instruct-NeRF2NeRF fails and we also produce higher diversity. (2) Our method allows for user-guided edits from key-view edits or even single-view edits, which is more convenient. (3) Our method also provides efficient benefits, as noted by the reviewer.  (4) As suggested by reviewer iE4L, we further provide an application of local editing using our method in **Fig. R2** of the rebuttal PDF, which is infeasible for previous works.\n\n> Q3. Keyframe selection criterion seems a little bit arbitrary\n\nA3. Our design of the keyframe selection criterion is not arbitrary; instead, it follows the rationale to ensure that the next selected frame maintains a suitable overlap with the previous keyframes. Intuitively, if the overlap is high every time, more keyframes will be sampled, which will affect the efficiency of our method. Conversely, if the overlap is low, the new edits will not take previous edits into account, resulting in inconsistent edits at the boundaries. Based on this rationale, we use a simple and straightforward piecewise linear function for sequential keyframe selection, where the largest weight is assigned to frames that exhibit a certain overlap. Emplically, we found that our keyframe selection criterion works well in our experiments.\n\n> Q4. How does the value of $\\phi$ affect the final result?\n\nA4. Per the reviewer’s request, we provide an ablation study on the hyperparameter $\\phi$ in **Fig. R6** of the rebuttal PDF. When it's too large, e.g.1, the frame with the largest overlap will be chosen. Thus it will result in low efficiency, but the overall result will be consistent. When it's too small, the frame with the lowest overlap will be chosen. In such cases, the modification may follow different textures and color, making inconsistent edits on the boundaries of edits.\n"}, {"review_id": "bBGN5HX8xq", "reviewer": "Reviewer_iE4L", "summary": "This paper introduced ViCA-NeRF, a method for 3D-consistent NeRF editing using diffusion models. The proposed method decouples NeRF editing with two steps. In first stage (i.e. dataset editing), key views are sampled and edited via Instruct-Pix2Pix model and propagated to other views using the NeRF depth maps, obtaining a mixup dataset. In second stage (NeRF training), the edited dataset can be directly used for training the NeRF model. The authors proposed  a blending refinement model to enhance the quality that enables consistent NeRF editing results.", "questions": "Some related works on 3D consistent NeRF editing:\n- PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields. CVPR 2023.\n- Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field. CVPR 2023.  \n- ARF: Artistic Radiance Fields. ECCV 2022. \n- SNeRF: Stylized Neural Implicit Representations for 3D Scenes. SIGGRAPH 2022.\n- NeRFshop: Interactive Editing of Neural Radiance Fields. I3D 2023.  ", "limitations": "Discussed in weakness section. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method demonstrated 3D consistent NeRF editing results and often achieves better editing image quality than baseline Instruct-NeRF2NeRF and NeRF-Art. Concretely, here are some strengths of the proposed method:\n- Guided Editing: The method uses key views for consistent and detailed 3D scene manipulation;\n- Depth-guided Correspondence Matching: Allows projecting one edited view onto all others for consistency and detail;\n- Key Frame Sampling and Mixup: This reduces the computational load by representing the entire scene with a few key frames; \n- Blending Refinement Model: Reduces noise in depth information and resolves incongruities between 2D edits and their 3D context;\n- Warm-Up and Post-Refinement Strategy: Enhances editing quality and efficiency;\n- Semantically Meaningful Edits: The method can make significant edits based on textual prompts.\n\nIn summary, the method is capable of performing 3D consistent NeRF editing with text-guided diffusion pipeline. The quality of the results demonstrate improvements over previous approaches, sufficient ablation and comparisons are shown. \n", "weaknesses": "There are some limitations and concerns. \n- Firstly, I was wondering what happens if the NeRF depth or geometry is bad, such as on glossy or specular surfaces. In such case, the depth-guided correspondence matching would decrease its accuracy. Considering most scenes shown in the paper are diffuse, it'd be interesting to have one such example as an ablation.\n- Results are rather blurry, such as Flowers in Fig.1, Cyberpunk Style in Fig. 4 and Fireworks in Fig. VI. In contrast, the \"original NeRF\" on the left exhibits pretty sharp image details. What's the source of blurriness --- Is it due to key frame selection, depth quality, or diffusion model? Some analysis and (at least) discussions are necesssary and helpful. \n- How's Equation (2) proposed? And the hyperparameter phi = 0.3. \n- Can local edits be enabled? Currently the pipeline is, similar to Instruct-NeRF2NeRF, heavily based on Instruct-Pix2Pix. This is typically diffusion on the whole image (i.e. background is also changing and being modified). Would be nice to support locally editing of a person without touching the background in some cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Below we address all the concerns.\n\n> Q1. What happens if the NeRF depth or geometry is bad, such as on glossy or specular surfaces\n\nA1. We investigate the robustness of our method against inaccuracy or failure in depth estimation in the rebuttal PDF. Per the reviewer’s request, we consider glossy surfaces, e.g. glass, which will cause incorrect depth. In this case, we are still able to propagate reasonable editing, as shown in **Figs. R3** of  the rebuttal PDF. Additionally, we evaluate the case with sparse view inputs, where the depth also becomes less accurate as the information of view consistency is reduced. **Figs. R5** of the rebuttal PDF shows that our method can still reach consistent edits as long as the NeRF is successfully trained.\n\nIn the most extreme case where the estimated depth is completely wrong, there is no correct correspondence between different frames. Thus, the single-view editing won’t be able to propagate to other views. Our method will degenerate into a view-independent editing strategy, where each frame is updated through Instruct-Pix2Pix. \n\n> Q2. What's the source of blurriness\n\nA2. The blurriness mainly comes from the diffusion model. As the mixed image provide guidance for diffusion, if there are lots of artifacts after mixup, some high-frequency information is lost to match the target description. We show this phenomenon in rebuttal PDF Figure R1, where the wall tends to be kind of blurred. After the blending process, some regions are smoothed. For possible reason of artifacts, besides the inaccurate depth, the major problem is from Instruct-Pix2Pix. It tends to  Irregularly changes the object boundary or background, resulting in artifacts when projected. Regarding this, future diffusion models with better 2D consistency may help to reduce this problem. \n\n> Q3. How's Equation (2) proposed? And the hyperparameter phi = 0.3.\n\nA3. Equation (2) represents our keyframe selection criterion. Our intuition for Equation (2) is that each time we want the next selected keyframe to have a suitable overlap with observed regions from the previous keyframes. If the overlap is high every time, more keyframes will be sampled, which will affect the efficiency of our method. Conversely, if the overlap is low, the new edits will not take previous edits into account, resulting in inconsistent edits at the boundaries. Thus, we design a simple and straightforward piecewise linear function to assign the largest weight to frames that exhibit a certain overlap. The hyperparameter $\\phi$ is chosen to balance efficiency and performance. When $\\phi$ is large, frames with a higher overlap are encouraged, resulting in lower efficiency. When $\\phi$is small, frames with a lower overlap are encouraged, which may cause inconsistent overlap regions. We show an ablation study on $\\phi$ in **Fig.R6**\n\n> Q4. Can local edits be enabled?\n\nA4. Yes. Since our model can benefit from frame propagation, it is easy to integrate local edits by propagating the local mask automatically. Here we show one example by using Segment Anything Model (SAM) [Ref1] for local editing. The results are shown in **Fig. R2** of the rebuttal PDF. We can achieve local edits without touching the background. \n\nSpecifically, we segment each image into regions by using SAM. Given a single view, the user indicates the object’s position, resulting in a segmentation mask. By propagating and mixing through depth, the highly overlapped regions in each frame are combined as the object mask. Then, only the masked regions are edited.\n\n> Q5: Some related works on 3D consistent NeRF editing\n\nA5: Thank you for your great suggestion! We will cite and discuss these papers in our revision. Below is a brief summary of comparisons between our work and these papers.\n\nPaletteNeRF: Rather than just changing the color, we manage to change the content according to text instructions.\n\nSINE: For real scenes, SINE focuses on changing the texture for a certain style or material, while our method focuses on general content editing. \n\nARF: ARF tries to match the feature space to one reference image, where the main purpose is style changing.\n\nSNeRF: SNeRF uses an image stylization model and trains NeRF with the updated image. Instruct-NeRF2NeRF shares a similar flow as this work. By contrast, our method tried to directly modify a view-consistent dataset, which is much more efficient.\n\nNeRFshop: NeRFshop mainly deals with geometry editing of NeRF, while our method focuses on the content perspective.\n\n[Ref1] Kirillov, Alexander, et al. \"Segment anything.\" arXiv preprint arXiv:2304.02643 (2023).\n"}, {"review_id": "x3xE8GvQCm", "reviewer": "Reviewer_CQtY", "summary": "The paper proposes a method to edit a NeRF using natural language. The authors achieve this by first “generating” an edited version of the input dataset and then training a NeRF on the edited dataset. To generate the said dataset, the authors train a NeRF on the original dataset for depth, with which they then propagate their keyframe edits (made with Instruct-Pix2Pix) to other views. Because NeRF depth is noisy, the authors use a diffusion-based blending network to refine those images into a cleaner dataset for the next-stage NeRF, which simply learns the radiance field from the new, edited images.\n", "questions": "Because I don’t expect the authors to change their method, I’m primarily interested in hearing about points #2 and #3 from the Weaknesses section above.\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper tackles the difficult problem of editing 3D scenes in a multi-view-consistent way. The proposed method is general, being applicable to humans, objects, and even unbounded scenes. It also takes advantage of physics-based operations such as projections and combines them with learning to handle noise and uncertainty (e.g., in NeRF depth). The paper produces plausible results (only up to a low resolution though).", "weaknesses": "\nThe major weakness, I think, is the complexity of the pipeline that requires the user to train a NeRF for depth, run inference on keyframes with Instruct Pix2pix, project edited frames to other views, run a diffusion model to perform blending, and finally train another NeRF on the edited images. It does not sound like an easy-to-use pipeline that can be flexibly extended or built upon. A complicated pipeline like this has many points of failure, too, as compared with a simpler pipeline.\n\nThe question then becomes which part of the pipeline we may cut to achieve similar results. The authors claim it’s crucial to perform diffusion-based blending to get around the artifacts due to direct mixup. Indeed, the results look quite blurry. Given (the 2nd-stage) NeRF also blurs the results, I think the authors should ablate the blending module, i.e., directly feeding the mixup output into the 2nd-stage NeRF. My hunch is NeRF’s blurring effect may already achieve what the additional diffusion model is doing, To be clear, I do think blurry results are a weakness of this paper, too.\n\nI think one crucial experiment on the number of edited keyframes is missing: how many keyframes does one need to edit to achieve good results? Why not just run Instruct Pix2pix on all views instead of doing mixup and then blending?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Below we address all the concerns.\n\n> Q1. Complexity of the pipeline that requires the user to train a NeRF for depth, run inference on keyframes with Instruct Pix2pix, project edited frames…\n\nA1. We respectfully disagree with the reviewer that our pipeline is more complicated and less flexible in terms of extensibility compared with prior works like Instruct-NeRF2NeRF. Instead, as noted by Reviewer x1ev, “The motivation of the proposed pipeline is sound and makes sense. It is also simple, which makes it easy to follow.” Below we clarify the simplicity and extensibility of our pipeline from several important aspects.\n\n(1) **Our pipeline shares the same usage complexity with Instruct-NeRF2NeRF**. Much like Instruct-NeRF2NeRF, users are only required to provide a text prompt for editing. Our entire pipeline operates automatically, without necessitating manual intervention to sequence or  control its components.\n\n(2) **Our pipeline is designed in a highly structured, modular manner**, with each component contributing to improve the result while maintaining a degree of independence (as shown in Fig. 7). This design allows us to easily improve individual components separately, making the pipeline flexible and extensible.\n\n(3) Critically, different from the iterative update strategy used in Instruct-NeRF2NeRF, **our method decouples the processes of NeRF finetuning and data editing**, enabling direct acquisition of the final edited data. Thus, our method is more flexible to fit different NeRF models. In addition, by getting rid of iterative updates which cause about 6 times as long as the regular NeRF training, our method is much faster compared with Instruct-NeRF2NeRF.\n\n(4) Finally, we would like to emphasize that a pre-trained NeRF is always needed for the NeRF editing task. The depth is extracted from the intermediate density value and **does not need to be separately trained**. The mixup and blending process are designed to propagate sparse view edits to dense view edits.\n\n> Q2. Which part of the pipeline we may cut to achieve similar results?\n\nA2. As shown in Fig. 7 of the main paper and Fig. I of the supplementary, we systematically ablated different components of our method, and validated that all our components contribute to the improvement of editing results. More specifically, (1) regarding the mixup model, it propagates the key view edits, thus promoting global consistency. (2) Regarding the blending model, there will exist noticeable artifacts if it is removed, as ablated in Figs. 7 and I. This will be more significant for larger-scale scenes Indeed, as pointed out by reviewer CQtY, “Blending Refinement Model: Reduces noise in-depth information and resolves incongruities between 2D edits and their 3D context”. (3) In addition to these two modules, the warm-up process is designed to shift the image domain to target description efficiently, and the post-refinement further improves the results.\n\nAs for the blurry results, it's mainly caused by the diffusion model. Different from Instruct-NeRF2NeRF where the blurriness is introduced by iterative updates, ours come from the artifacts. As shown in **Fig.R1**, when there are too many artifacts, some parts, e.g. the wall, will be smoothed. Thus, the final result will be blurred. We think this can be solved by a better 2D consistent diffusion model in the future. In addition, using the local editing strategy can directly get rid of the background-changing problem. \n\n> Q3. How many keyframes does one need to edit to achieve good results?\n\nA3. The number of keyframes depends on the scale of the scene, specifically, the range of camera movement. As each time we sample a keyframe until all the views are covered by a minimum ratio, more keyframes will be sampled on larger scenes. However, the whole editing process is automatic based on the input text prompt, similar to Instruct-NeRF2NeRF. Typically, for the face scene with 60 views, 7 key views will be taken.\n\n> Q4. Why not just run Instruct Pix2pix on all views instead of doing mixup and then blending?\n\nA4. Simply updating all views will lead to inconsistent edits and broken results. We did not discuss such a strategy, since it has been already ablated in previous work Instruct-NeRF2NeRF – Figure 8 “One time Update” in Instruct-NeRF2NeRF. Their observation is: “the initial edited 2D images are largely inconsistent, leading to blurry and artifact-filled 3D scenes.”\n"}, {"review_id": "1IFjnbdLXM", "reviewer": "Reviewer_6ncD", "summary": "This paper addresses the task of editing 3D scenes using text prompt. The method is based on performing 2D edits on different views using a diffusion model and then optimizing the 3D representation on the edited images. In particular, the authors propose two regularization methods to ensure the 3D consistency of the 2D edits: geometric regularization based on depth-aware warping of of the edited images to other views, and learned regularization by aligning the latent codes in the diffusion model. The proposed method is compared to two main baselines for editing of several 3D scenes.", "questions": "-- I did not fully understand how the key frames are selected, and how many key frames are used in the proposed method. I would appreciate if authors could further clarify that.\n\n-- Are the key frames edited independently and then propagated to non-key frame views, or are the key frames also edited sequentially using the previously edited ones? \n\n-- Is there any ordering when the edited key-frames warped to other views sequentially, or is the key frames are selected randomly and independently?\n\n-- Some of the limitations of the method are discussed in the limitation section. It would have been more helpful to also include some visualizations of the failure cases, especially the ones which do not happen in IN2N.\n", "limitations": "The limitations and social impact have been addressed by the authors.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "-- Addressing the 3D consistency of text-based 3D editing is valuable, and is currently missing from the existing methods.\n\n-- The proposed method has sufficient novelty containing different new components compared to the baseline. \n\n-- The proposed regularization based on depth-aware warping is reasonable.\n\n--  Not requiring the iterative editing and optimization used in the main baseline, Instruct-NeRF-to-NeRF (IN2N), enables a more efficient editing (3x faster based on what authors report).\n\n-- The experiments show more promising results than the baseline in terms of 3D consistency.", "weaknesses": "-- I am mainly not convinced about the visual quality proposed methods, as the visual results seem to be quite blurry and lack details. Although IN2N also suffers from some blurry results, it seems the problem is intensified in the proposed method due to the averaging of the latent codes. I am not sure if the proposed averaging is a good solution, considering the blurry outcomes.\n\n-- The method section is not well-written and it is quite hard to follow. I find some details of the proposed method to be either missing or not well-explained. Also, the diagrams visualizing the method are too general and not helpful enough in understanding the components of the proposed method. I am afraid the current representation of the paper might not be ready for publication. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Below we address all the concerns.\n\n> Q1. it seems the burry problem is intensified in the proposed method due to the averaging of the latent codes\n\nA1. According to our experiments, the averaging process won’t make more blur results as shown in **Fig. R1** of the rebuttal PDF.  The main reason causing this blurry issue is the diffusion model. As our method wants to fully leverage the power of 2D editing,  As Instruct-Pix2Pix often changes borders and backgrounds erratically, the extensive 2D edits will result in artifacts for other views. When blending the image, though artifacts will be removed, the region will become smoother since some high-frequency information is unrecognizable. Thus, the result tends to have blurriness. Potentially, this can be improved from future 2D consistent diffusion models.\n\nFor the question regarding \"lack of details\", we want to emphasize that we can actually make correct details for edits that are harder, e.g. spider man mask, Edward Munch painting, robot bear and other results shown.\n\n> Q2. How the keyframes are selected, and how many keyframes are used?\n\nA2. The number of keyframes depends on the scale of the scene, specifically, the range of camera movement. We sequentially sample keyframes until all views are covered by keyframes from a certain ratio. To sample the next keyframe, we calculate the covered ratio for each view and sort their weight to choose the frame with the largest weight. In practice, for the face scene with 60 views, 7 key views are sampled. However, the overall appearance is dominated by the first keyframe, e.g. **Figure I** in supplementary\n\n> Q3. Are the keyframes also edited sequentially using the previously edited ones?\n\nA3. Yes, they are sequentially edited. Though we allow automatic editing with text prompt, we also want to enable the user to guide the edits by key view editing, e.g. single view edits. As shown in Fig. I, we can control the whole edit with a single view.\n\n> Q4. Some visualizations of failure cases\n\nA4. Thank you for your suggestion. We include some hard cases in **Figs. R3 and R5** of the rebuttal PDF. Specifically, we showcase the results when the depth is worse, due to glossy surfaces or sparse view inputs. The glossy surface, e.g. glass, will cause incorrect depth. For sparse views, as the information of view consistency is reduced, the depth will become less accurate. While these challenges lead to worse performance, our approach can still make reasonable edits. Importantly, in contrast to previous methods like Instruct-NeRF2NeRF, which could experience editing failures in multiple scenarios, our approach is always able to edit the scene as long as the 2D diffusion model remains functional. However, the editing quality still depends on the quality of the NeRF representation. This also means that we did not encounter cases where our approach fails in scenarios that do not occur in Instruct-NeRF2NeRF.\n\n> Q5. Presentation of the paper: some details of the proposed method to be either missing or not well-explained\n\nA5. We hope that our detailed explanation and clarification provided above have addressed the reviewer’s concern. If any particular aspect remains unclear, we are more than happy to offer further elaboration in the discussion phase. Furthermore, we will improve the clarity of the revised paper, and will modify the methodology diagram to incorporate additional details for better understanding.\n"}]

Author rebuttal:
We thank all reviewers for their time and feedback. We are excited to read that Reviewer x1ev appreciated the quality of results and the rationality of the method; that Reviewer EWGc emphasized the motivation, efficiency of our method, and the clearness of the presentation; that Reviewer iE4L praised the validity and merits of different parts of our approach and provide inspiring related work; that Reviewer CQtY complimented that our method can work on various scenes; that Reviewer 6nCV underlined the novelty and promising results of our method. We address individual questions in replies to each review.

Based on your great suggestions, we made additional studies on our method. The results are shown in the PDF, and a brief summary is as follows:

* For the blurriness, we provide an explanation for such a phenomenon.

* We provide an application of local editing using our method with Segment Anything Model. Edits can be done for certain objects without changing the background.

* A user study regarding diversity and quality is provided.

* An ablation study on input views is conducted.

* We validate the effectiveness on glossy surfaces and noisy depth.

* Ablation studies on sparse input views.

* Additional editing results




Meta-review: The reviewers appreciated the paper’s contribution of addressing the issue of inconsistency across different views when performing NeRF edits and its demonstrated improvement over prior baselines. Post rebuttal and discussion, there was some remaining concern regarding the blurriness of the results. Nonetheless, the overall contribution and improvement over prior work outweighs this concern. Please take into account all reviewer feedback in the camera-ready version.
Decision: Accept (poster)