Title: ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural radiance field

diffusion model

editing
Abstract: We introduce ViCA-NeRF, the *first* view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that *explicitly* propagate the editing information across different views, thus ensuring multi-view consistency. For *geometric regularization*, we leverage the depth information derived from NeRF to establish image correspondences between different views. For *learned regularization*, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is available at: https://github.com/Dongjiahua/VICA-NeRF

ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields

Abstract

We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization , we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization , we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is available at: https://github.com/Dongjiahua/VICA-NeRF.

1 Introduction

The recent advancements in 3D reconstruction technology, exemplified by the neural radiance field (NeRF) [1] and its variants, have significantly improved the convenience of collecting real-world

3D data. Such progress has opened up new opportunities for the development of various 3D-based applications. By capturing RGB data from multiple views and obtaining corresponding camera parameters, NeRF enables efficient 3D representation and rendering from any viewpoint. As the availability of diverse 3D data increases, so does the demand for 3D editing capabilities.

However, performing 3D editing on NeRF is not straightforward. The implicit representation of NeRF makes it challenging to directly modify the 3D scene. On the other hand, since NeRF is trained on RGB images, it can effectively leverage a wealth of existing 2D models. Motivated by this, state-of-the-art methods, such as Instruct-NeRF2NeRF [2], apply 2D editing on individual images via a text-instructed diffusion model (e.g., Instruct-Pix2Pix [3]), and then rely exclusively on NeRF to learn from the updated dataset with edited images and propagate the editing information across different views. Despite yielding encouraging results, this indirect 3D editing strategy is time-consuming, because of its iterative process that requires cyclically updating the image dataset while progressively fine-tuning NeRF parameters. And more seriously, due to the lack of inherent 3D structure, the editing results from the 2D diffusion model often exhibit significant variations across different views. This raises a critical concern regarding 3D inconsistency when employing such a strategy, particularly in cases where the editing operation is aggressive, e.g., 'Add some flowers' and 'Turn him into iron man' as illustrated in Figure 1.

To overcome these limitations, in this work we introduce ViCA-NeRF, a View-Consistency-Aware NeRF editing method. Our key insight is to exploit two additional sources of regularization that explicitly connect the editing status across different views and correspondingly propagate the editing information from edited to unedited views, thus ensuring their multi-view consistency. The first one is geometric regularization . Given that depth information can be obtained from NeRF for free, we leverage it as a guidance to establish image correspondences between different views and then project edited pixels in the edited image to the corresponding pixels in other views, thereby enhancing consistency. This approach allows us to complete preliminary data modifications before fine-tuning NeRF, as well as enabling users to determine 3D content by selecting the editing results from key 2D views. Moreover, as our method allows for direct and consistent edits, it is more efficient by avoiding the need for iterative use of the diffusion model.

The second one is learned regularization . The depth information derived from NeRF can often be noisy, yielding certain incorrect correspondences. To this end, we further align the latent codes in the 2D diffusion model (e.g., Instruct-Pix2Pix) between edited and unedited images via a blending refinement model. Doing so updates the dataset with more view-consistent and homogeneous images, thereby facilitating NeRF optimization.

Our ViCA-NeRF framework then operates in two stages, with these two crucial regularization strategies integrated in the first stage. Here, we edit key views and blend such information into every viewpoint. To achieve this, we employ the diffusion model for blending after a simple projection mixup. Afterward, the edited images can be used directly for training. An efficient warm-up strategy is also proposed to adjust the editing scale. In the second stage, which is the NeRF training phase, we further adjust the dataset. Our method is evaluated on various scenes and text prompts, ranging from faces to outdoor scenes. We also ablate different components in our method and compare them with previous work [2]. Experimental results show that ViCA-NeRF achieves higher levels of consistency and detail, compared with the state of the art.

Our contributions are three-fold. (1) We propose ViCA-NeRF, the first work that explicitly enforces multi-view consistency in 3D editing tasks. (2) We introduce geometric and learned regularization, making 3D editing more flexible and controllable by editing key views. (3) Our method significantly improves the efficiency of 3D editing, achieving a speed 3 times faster than the state of the art.

2 Related Work

Text-to-Image Diffusion Models for 2D Editing. Diffusion models have recently demonstrated significant capabilities in text-to-image editing tasks [3-8]. Earlier studies have leveraged pre-trained text-to-image diffusion models for image editing. For instance, SDEdit [8] introduces noise into an input image and subsequently denoises it through a diffusion model. Despite achieving reasonable results, this approach tends to lose some of the original image information. Other studies have concentrated on local inpainting using provided masks [4, 5], enabling the generation of corresponding content within the masked area with text guidance. More recently, Instruct-Pix2Pix [3] proposes an

Instruct-Pix2Pix

Text: Turn him into

Depth

Mixup

Blending

Original Images

Stage1: Dataset Editing

Source NeRF

approach to editing images based on text instructions. Instruct-Pix2Pix accepts original images and text prompts as conditions. Trained on datasets of editing instructions and image descriptions, this approach can effectively capture consistent information from the original image while adhering to detailed instructions.

In this paper, we adopt Instruct-Pix2Pix as our editing model. Importantly, we further explore the potential of Instruct-Pix2Pix by extending its application to editing, blending, and refinement tasks in more challenging 3D editing scenarios.

Implicit 3D Representation. Recently, neural radiance field (NeRF) and its variants have demonstrated significant potential in the field of 3D modeling [1, 9-14]. Unlike explicit 3D representations such as point clouds or voxels, NeRF uses an implicit representation to capture highly detailed scene geometry and appearance. While training a conventional NeRF model can be time-consuming, InstantNGP [10] accelerates this process by training a multi-layer perceptron (MLP) with multi-resolution hash input encoding. Notably, NeRFStudio [9] offers a comprehensive framework that can handle various types of data. It modularizes each component of NeRF, thereby supporting a broad range of data. These efforts bridge the gap between image data and 3D representation, making it possible to guide 3D generation from 2D models. Our work uses the 'nerfacto' model from NeRFStudio to fit real scenes. We also leverage the depth estimated from the model to maintain consistency.

3D Generation. Inspired by the success of 2D diffusion models [15, 16], 3D generation has drawn great attention in recent years [17-21]. Dream Fusion first proposes to use a score distillation sampling (SDS) loss as the training target. Later, Magic3D [20] generates higher resolution results by a coarse-to-fine procedure. Various scenarios have been explored, including generating a detailed mesh from text prompts [21] and generating novel objects based on images from a few views [18].

Rather than generating new content, our work focuses on editing the 3D representation with text prompts. By explicitly leveraging 2D editing, we provide a more direct approach to control the final 3D result.

NeRF Editing. Because of the implicit representation of NeRF [1], editing NeRF remains a challenge. Early attempts [22-32] perform simple editing on shape and color, by using the guidance from segmentation masks, text prompts, etc. NeuralEditor [33] proposes to exploit a point cloud for editing and obtain the corresponding NeRF representation. Another line of work tries to modify the content through text prompts directly. NeRF-Art [34] uses a pre-trained CLIP [35] model to serve as an additional stylized target for the NeRF model. More recently, Instruct-NeRF2NeRF [2] and Instrcut-3D-to-3D [36] leverage Instruct-Pix2Pix [3] for editing. They can edit the content of NeRF with language instructions. However, both require using the diffusion model iteratively in the NeRF training process, making them less efficient. More importantly, though it shows promising results on real-world data, Instruct-NeRF2NeRF can only work when the 2D diffusion model produces consistent results in 3D. Thus, it is unable to generate a detailed result where 2D edits tend to be different, and it also suffers from making diverse results from the same text prompt.

Text: Turn him into

Iron Man

Original Images

Diffusion

Process

Diffusion

Process

Diffusion

Process

Mean

E

D

Text Cond

Image Cond

Diffusion

Process

Diffusion

Process

Diffusion

Process

Mean

E

D

Text Cond

Image Cond

Blended Image

Image Mixup

Blending Model

Modified Inp2p

with average

Projection

Mixup

Edited

Origin

Depth

In our work, we propose to use depth guidance for detailed 3D editing. Our ViCA-NeRF only needs to update the dataset at the start of training and further refine it once, making it significantly more efficient. In addition, our method results in more detailed and diverse edit results compared with Instruct-NeRF2NeRF. As several key views guide the whole 3D scene, we allow users to personalize the final effect on NeRF by choosing 2D edit results.

3 Method

Our ViCA-NeRF framework is shown in Figure 2. The main idea is to use key views to guide the editing of other views, ensuring better consistency and detail. Given the source NeRF, we first extract each view's depth from it and sample a set of key views. Then, we edit these key views using Instruct-Pix2Pix [3] and propagate the edit result to other views. To address the problem of incorrect depth estimation from NeRF, we perform mixup on each image and then refine it through our blending model. The obtained data can be used directly for NeRF training. Optionally, we can perform post-refinement to the dataset after obtaining more consistent rendering results from NeRF.

3.1 Preliminary

We first elaborate on NeRF, including how we obtain rendered RGB images and coarse depth from the NeRF model. Subsequently, we present Instruct-NeRF2NeRF, a diffusion model for 2D editing. With this model, we can perform 2D editing using text and then further enable 3D editing.

Neural Radiance Field. NeRF [1] and its variants [9, 10] are proposed for 3D scene representation. The approach models a 3D scene as a 5D continuous function, mapping spatial coordinate ( x, y, z ) and viewing direction ( θ, ϕ ) to volume density σ and emitted radiance c . For a given set of 2D images, NeRF is trained to reproduce the observed images. This is achieved by volume rendering, where rays are traced through the scene and colors are accumulated along each ray to form the final rendered image. While NeRF does not explicitly predict depth, one common approach is to compute a weighted average of the distance values along each ray, where the weights are the predicted volume densities. This process can be conceptualized as approximating the depth along the ray.

Instruct-Pix2Pix. Instruct-Pix2Pix [3] is a 2D diffusion model for image editing. It takes a text prompt c T and a guidance image c I as conditions. Given an image z 0 , it first adds noise to the image to create z t , where t is the timestep to start denoising. With denoising U-Net ϵ θ , the predicted noise can be calculated as

When a predicted noise is calculated, Instruct-Pix2Pix follows the standard denoising process to update the image z t step by step and obtain the final result z 0 , which is an edited image consistent with the initial image and text instruction.

'Turn the bear into a grizzly bear'

'Make it surrounded by flowers'

Original NeRF

Instruct-NeRF2NERF

Ours

Instruct-NeRF2NERF

Ours

'Turn the bear into a robot bear'

'Make it Cyberpunk Style'

'Turn him into a robot'

'Turn him into Spider man with mask'

'Add some flowers'

'Make it looks like snowed'

3.2 Depth-Guided Sequential Editing

In our approach, we choose to reflect the user's 2D edits in 3D. Thus, we propose a sequential editing strategy. By defining certain key views, we sequentially edit them by projecting the previous edits to the next key view. This is the geometric regularization that explicitly enforces the consistency between different views.

Sequential Editing with Mixup. The overall procedure of editing is simple yet effective, which consists of editing key views and projecting to other views. First, we select a sequence of key views starting from a random view. Then, we sequentially use Instruct-Pix2Pix to edit each view and project the modification to the following views. At the same time, non-key views are also modified by projection. After editing all key views, the entire dataset is edited.

Given that we project several key views onto other views when they are edited, each view could be modified multiple times. Only pixels that have not yet been modified are altered to prevent inadvertent overwriting. Therefore, modifications from key views are mixed up with each view.

Consequently, our method yields a final result that aligns closely with the modified views, facilitating a controllable editing process. Since previously modified pixels are not changed until blending refinement (explained in Section 3.3), the edits of the first few views can dominate the final result.

Key View Selection. We employ key views to make the editing process consistent. This strategy allows the entire scene to be constrained by a handful of edited views. To ensure the edited views maintain consistency and bear similar content, we define the modified ratio ρ for each view as the fraction of modified pixels. Then, we use the following equation to select the view with the largest

weight w as the next key view:

where ϕ signifies the maximum desired ratio, which is set to ϕ = 0 . 3 in all experiments. Intuitively, each time the next selected key view will have a suitable overlap with the edited region from each of the previous key views. The key view selection ends when all views' minimum ρ exceeds a certain threshold.

Depth-Guided Projection. Modern NeRF models are capable of generating depth maps directly, thereby making it possible to construct geometric relationships between different views. We illustrate an example of projecting from edited view j to view i as below.

Given an RGB image I i and its corresponding estimated depth map D i , the 3D point cloud associated with view i can be calculated as:

Here, K i denotes the intrinsic parameters, ξ i represents the extrinsic parameters, and P -1 is a function mapping 2D coordinates to a 3D point cloud.

To find the corresponding pixels on another view, we project this point cloud to view j as follows:

where I i,j signifies the color of the pixels projected from I j onto I i . In practice, to ensure the correct correspondence and mitigate the effects of occlusion, we compute the reprojection error to filter out invalid pairs. By using this correspondence, we can project pixels from view j to view i .

3.3 Blending Refinement Model

The depth information derived from NeRF [1] can often be quite noisy, resulting in noticeable outliers, so only applying the geometric regularization is insufficient. Specifically, while we can filter out incorrect correspondences, the remaining pixels and projections from varying lighting conditions may still introduce artifacts. Furthermore, a reasonable 2D edit in a single view may appear strange when considered within the overall 3D shape. To address these challenges, we introduce a blending refinement model that employs two Instruct-Pix2Pix [3] processes, acting as the learned regularization to improve the overall quality.

Details about the model are shown in Figure 3, where each diffusion process serves unique functions. During the first pass, we use the mixup image as input and the original image as a condition. This procedure aims to purify the noisy mixup image by preserving the structural integrity of the original image. The resulting image appears dissimilar to the consistent mixup image but without any noise. Thus during the second pass, we use the resulting image from the first pass as input and the mixup image as the condition. Given the shared semantic information between these two images, the outcome strives to closely align with the mixup image's detailed characteristics. Doing so allows us to swiftly generate a clean, stylized result, bypassing the need for iterative updates to the dataset during training.

An essential component of our refinement model involves using an average latent code for each of the two processes. We discovered that the diffusion model tends to generate diverse results, making it challenging to closely align with the target structure or content. However, this effect can be substantially mitigated by averaging multiple runs. Specifically, we introduce n r noised latent codes and update them independently. Finally, we compute the average of the latent codes and decode it back into the image. This method provides a more stable and consistent output, enhancing the overall quality of the editing process.

3.4 Warm-Up and Post-Refinement Strategies

To make the editing more efficient and further improve the quality, we propose a warm-up strategy along with a post-refinement strategy.

Warm-Up. When editing via instruct-Pix2Pix, the extent of editing changes can be modulated through hyperparameters, such as the guidance scale [15]. However, achieving the desired overall

'Tolkien Elf'

'Fauvism'

'Lord Voldemort'

'Edvard Munch'

Gogh'

'Turn him into the Tolkien Elf'

'Make him look like a Fauvism

painting'

'Turn him into Lord Voldemort'

'Make him look like an Edvard

Munch painting'

ViCA-NeRF (

)

NeRF-Art

Instruct-NeRF2NeRF

Ours

style remains a problem. Although Instruct-NeRF2NeRF ensures consistent stylization, it necessitates iterative updates, which could inefficiently involve a large number of iterations.

To overcome these limitations, we introduce a strategy to efficiently warm up the editing process by blending edits directly. This approach is performed before the entire editing process. Specifically, we randomly select a view to edit, and then project the edited view onto all other views. For a given view i and an edited view j , an update is calculated as follows:

Here, m w denotes the binary mask for the projected area, and λ is a hyperparameter employed to control the ratio preserved from the original pixel value. This efficient warm-up strategy accelerates the editing process while preserving a high degree of consistency in the stylization.

Post-Refinement. To further enhance the consistency of the edited results, we employ a postrefinement strategy once the NeRF has been trained on our updated data. We still utilize the modified Instruct-Pix2Pix with averaging architecture to generate a more accurate target for each view. Unlike the first stage, this time we employ the rendered image as input while continuing to use the mixup image as a condition. This post-refinement strategy further refines the consistency and quality of the final output, contributing to a more cohesive and visually appealing result.

4 Experiments

We conduct experiments on various scenes and text prompts. All our experiments are based on real scenes with NeRFStudio [9]. We first show some qualitative results and comparisons between our method and Instruct-NeRF2NeRF [2]. For artistic stylization, we test our method with the cases from NeRF-Art [34]. Experiments show that we achieve more detailed edits. Based on these scenes, we further conduct ablation studies on our method, including the effects of different components in the framework, the warm-up strategy, failure cases, and representative hyperparameters. We also conduct quantitative evaluation of the results, testing the textual alignment, consistency, and efficiency of our method.

4.1 Implementation Details

For a fair comparison, our method employs a configuration similar to that of Instruct-NeRF2NeRF. We utilize the 'nerfacto' model from NeRFStudio and Instruct-Pix2Pix [3] as our 2D editing model. For the diffusion model, we apply distinct hyperparameters for different components. During the editing of key views, the input timestep t is set within the range [0.5, 0.9]. We employ 10 diffusion steps for this phase. In the blending refinement model, we set t to 0.6 and n r to 5, and use only 3 diffusion steps. The reduced number of steps enables the model to operate more quickly, since encoding and decoding are executed only once. The diffusion model provides additional adjustable

Figure 6: Different 2D edits and corresponding 3D edits. Our ViCA-NeRF achieves high correspondences between 2D edits and 3D edits, thus enabling users to control the final edits. On the other hand, our method exhibits high diversity with different random seeds.

Figure 7: Image update through our ViCA-NeRF. Our blending model effectively eliminates artifacts and boundaries caused by mixup. The post-refinement step further improves the consistency with the mixup image and the 3D content.

hyperparameters, such as the image guidance scale S I and the text guidance scale S T . We adopt the default values of S I = 1 . 5 and S T = 7 . 5 from the model without manual adjustment.

For training the NeRF, we use an L1 and LPIPS (Learned Perceptual Image Patch Similarity) loss throughout the process. Initially, we pre-train the model for 30,000 iterations following the 'nerfacto' configuration. Subsequently, we continue to train the model using our proposed method. The optional post-refinement process occurs at 35,000 iterations. We obtain the final results at 40,000 iterations.

4.2 Qualitative Evaluation

Our qualitative results, illustrated in Figure 4 and Figure 5, demonstrate the effectiveness of our method in editing scenes through multiple views. We present results on three scenes with varying complexity, from human figures to large outdoor scenes. We also evaluate the stylization capabilities of NeRF-Art [34] and compare our results with the previous state of the art.

For comparison with Instruct-NeRF2NeRF, we replicate some cases from their work and assess several more challenging cases. For simpler tasks such as 'turn the bear into a grizzly bear' or 'make it look like it has snowed,' our method achieves results similar to those of Instruct-NeRF2NeRF. This may be due to the inherent consistency of diffusion models in handling such scenarios. For more complex tasks that require additional detail, like transforming an object into a robot or adding flowers, our method shows significant improvements. A notable example is the 'Spiderman cas,' where Instruct-NeRF2NeRF struggles with differentiating lines on the mask, while our method renders a clear appearance.

When compared on NeRF-Art, our method clearly outperforms the previous techniques by exhibiting more details and adhering more closely to the desired style. Although we use the same 2D editing model as Instruct-NeRF2NeRF, we are able to produce more vivid colors and details, which are not achievable with iterative updates.

Moreover, a significant advantage of our method is that the final 3D edits can be directed through several key views. This allows us to achieve consistent edits not only across different 3D views but also between the 2D edits and the 3D renders, as shown in Figure 6.

4.3 Ablation Study

In our ablation study, we first evaluate the individual effects of mixup, blending, and post-refinement on the quality of editing. Further investigation includes the impact of warm-up iterations and NeRF

Origin Image warmup=0

warmup=10

depth estimation accuracy. We also explore the influence of the parameter ϕ on key view selection for efficient editing. Each of these components is crucial for understanding the nuances of our method's performance.

Model Component Ablation. As shown in Figure 7, the mixup image may initially appear very noisy and contain artifacts. When we apply our blending model to refine it, the boundary from copy-pasting and artifacts is removed. However, while maintaining almost all the original content, the appearance of the image still changes slightly. Finally, when we apply our post-refinement strategy to update it, the result improves and becomes similar to the rendered result. Notice that the result after blending is sufficiently good, making further refinement an optional step.

Warm-Up Iteration Ablation. Considering that Instruct-Pix2Pix relies heavily on the input image, modifications to the input can significantly affect the output distribution. Therefore, to validate our warm-up procedure, which is designed to alter the scale of editing, we adjust this parameter from 0 (no warm-up) to 30. As Figure 8 illustrates, our findings indicate that while preserving the primary geometry, our warm-up procedure can effectively act as a controller for adjusting the scale of editing, introducing only minor computational costs.

Ablation for Depth Estimation Errors. The projection procedure is critical in our framework; therefore, inferior depth estimation may affect the final result. We conduct an ablation study focusing on two prevalent scenarios: glossy surfaces and sparser views. As Figure 9a demonstrates, our model can accurately make modifications to the bookshelf. If the reprojection check is omitted, more extensive changes are possible, albeit at the cost of losing original details. With sparser views, as depicted in Figure 9b, our model is capable of editing provided that the NeRF is successfully trained.

Ablation on Key Views. We perform experiments based on the hyperparameter ϕ . Specifically, we test the results with ϕ set to 0, 0.3, and 1, as illustrated in Figure 10. When ϕ = 0 , the view with the least overlap is selected, resulting in a clear boundary (highlighted in the red box). In other cases, this issue is mitigated. However, when ϕ = 1 , nearly all views are designated as key views, which leads to reduced efficiency.

4.4 Local Editing

Existing methods often induce simultaneous changes in both background and foreground during editing, which contradicts specific user intentions that typically focus on altering distinct objects.

Turn the shelf into a metal shelf

Original Data

Trained NeRF

ViCA-NeRF (Ours)

Ours (w/o reprojection check)

20%

(13) views

50%

(59) views

50%

(33) views

'Turn his face in to a skull'

Original

ViCA-NeRF (Ours)

(a)

Results with glossy surface

(b)

Ablation study on input views

� = 1

� = 0.3

� = 0

'Turn him into spider man'

propogate

(a) Local mask propagation w/o local edit

'Turn his face in to a skull'

(b) Local editing

To address this discrepancy, we introduce a straightforward, user-friendly local editing technique within our framework. Initially, we employ Segment Anything Thing (SAM) [37] to derive masks encompassing all components within each image. Subsequently, users pinpoint the target instance with a single point to isolate it. Following this, we utilize the depth information to project the identified mask across different views progressively. This process continues as long as the projected mask sufficiently overlaps with the corresponding mask in the current view. Then, we integrate the wrapped segmentation mask with the segmented masks in the current view.

Figure 11 exemplifies the process, resulting in a multi-view consistent segmentation that permits exclusive modification of the human figure, leaving the background intact.

4.5 Discussion

For quantitative evaluation, we adopt the directional score metric utilized in Instruct-Pix2Pix [3] and the temporal consistency loss employed in Instruct-NeRF2NeRF [2]. Due to the undisclosed split and text prompts from Instruct-NeRF2NeRF, we establish various difficulty levels to assess performance. Results and comprehensive details are provided in the supplementary material.

A further benefit of our approach is the decoupling of the NeRF training from the dataset update process. This separation significantly reduces the computational burden typically imposed by the diffusion model at each iteration. The only computational expense arises from the warm-up, blending, and post-refinement stages. We analyze the time cost of the 'Face' scene and compare it with Instruct-NeRF2NeRF on an RTX 4090 GPU. Both methods undergo 10,000 iterations; however, Instruct-NeRF2NeRF requires approximately 45 minutes, whereas our ViCA-NeRF only needs 15 minutes.

Limitations. Our method's efficacy is contingent upon the depth map accuracy derived from NeRF, which underscores our reliance on the quality of NeRF-generated results. Although our blending refinement can mitigate the impact of incorrect correspondences to some extent, the quality of the results deteriorates with inaccurate depth information. We also note that akin to Instruct-NeRF2NeRF, the edited outcomes tend to exhibit increased blurriness relative to the original NeRF. We investigate this phenomenon further and provide a detailed analysis in the supplementary material.

5 Conclusion

In this paper, we have proposed ViCA-NeRF, a view-consistency-aware 3D editing framework for text-guided NeRF editing. Given a text instruction, we can edit the NeRF with high efficiency. In addition to simple tasks like human stylization and weather changes, we support context-related operations such as 'add some flowers' and editing highly detailed textures. Our method outperforms several baselines on a wide spectrum of scenes and text prompts. In the future, we will continue to improve the controllability and realism of 3D editing.

/

trained NeRF

w

local edit

References

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021. 1, 3, 4, 6

Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-NeRF2NeRF: Editing 3D scenes with instructions. In ICCV , 2023. 2, 3, 7, 10

Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In CVPR , 2023. 2, 3, 4, 6, 7, 10

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR , 2022. 2

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , 2022. 2

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR , 2023.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR , 2023.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022. 2

Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In SIGGRAPH , 2023. 3, 4, 7

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 41(4):102:1-102:15, 2022. 3, 4

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR , 2020.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV , 2021.

Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR , 2021.

Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, and Noah Snavely. Neural 3D reconstruction in the wild. In SIGGRAPH , 2022. 3

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022. 3, 6

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021. 3

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation. In ICCV , 2023. 3

Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Ben Mildenhall, Nataniel Ruiz, Shiran Zada, Kfir Aberman, Michael Rubenstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. Dreambooth3D: Subject-driven text-to-3D generation. ICCV , 2023. 3

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In ICLR , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3D: High-resolution text-to-3D content creation. In CVPR , 2023. 3

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2Mesh: Text-driven neural stylization for meshes. In CVPR , 2022. 3

Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In ICCV , 2021. 3

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. In NeurIPS , 2022.

Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In ICCV , 2021.

Pei-Ze Chiang, Meng-Shiun Tsai, Hung-Yu Tseng, Wei-Sheng Lai, and Wei-Chen Chiu. Stylizing 3D scene via implicit representation and hypernetwork. In WACV , 2022.

Hsin-Ping Huang, Hung-Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan Yang. Learning to stylize novel views. In ICCV , 2021.

Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, and Lin Gao. StylizedNeRF: Consistent 3D scene stylization as stylized NeRF via 2D-3D mutual learning. In CVPR , 2022.

Qiling Wu, Jianchao Tan, and Kun Xu. PaletteNeRF: Palette-based color editing for NeRFs. In CVPR , 2023.

Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. SINE: Semantic-driven image-based NeRF editing with prior-guided editing field. In CVPR , 2023.

Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. ARF: Artistic radiance fields. In ECCV , 2022.

Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. SNeRF: Stylized neural implicit representations for 3D scenes. In WACV , 2022.

Clément Jambon, Bernhard Kerbl, Georgios Kopanas, Stavros Diolatzis, Thomas Leimkühler, and George Drettakis. NeRFshop: Interactive editing of neural radiance fields. Proceedings of the ACM on Computer Graphics and Interactive Techniques , 6(1), 2023. 3

Jun-Kun Chen, Jipeng Lyu, and Yu-Xiong Wang. NeuralEditor: Editing neural radiance fields via manipulating point clouds. In CVPR , 2023. 3

Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics , pages 1-15, 2023. 3, 7, 8

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021. 3

Hiromichi Kamata, Yuiko Sakuma, Akio Hayakawa, Masato Ishii, and Takuya Narihira. Instruct 3D-to-3D: Text instruction guided 3D-to-3D conversion. arXiv preprint arXiv:2303.15780 , 2023. 3

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV , 2023. 10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of editing 3D scenes using text prompt. The method is based on performing 2D edits on different views using a diffusion model and then optimizing the 3D representation on the edited images. In particular, the authors propose two regularization methods to ensure the 3D consistency of the 2D edits: geometric regularization based on depth-aware warping of of the edited images to other views, and learned regularization by aligning the latent codes in the diffusion model. The proposed method is compared to two main baselines for editing of several 3D scenes.\n\nSTRENGTHS:\n-- Addressing the 3D consistency of text-based 3D editing is valuable, and is currently missing from the existing methods.\n\n-- The proposed method has sufficient novelty containing different new components compared to the baseline. \n\n-- The proposed regularization based on depth-aware warping is reasonable.\n\n--  Not requiring the iterative editing and optimization used in the main baseline, Instruct-NeRF-to-NeRF (IN2N), enables a more efficient editing (3x faster based on what authors report).\n\n-- The experiments show more promising results than the baseline in terms of 3D consistency.\n\nWEAKNESSES:\n-- I am mainly not convinced about the visual quality proposed methods, as the visual results seem to be quite blurry and lack details. Although IN2N also suffers from some blurry results, it seems the problem is intensified in the proposed method due to the averaging of the latent codes. I am not sure if the proposed averaging is a good solution, considering the blurry outcomes.\n\n-- The method section is not well-written and it is quite hard to follow. I find some details of the proposed method to be either missing or not well-explained. Also, the diagrams visualizing the method are too general and not helpful enough in understanding the components of the proposed method. I am afraid the current representation of the paper might not be ready for publication.\n\nQUESTIONS:\n-- I did not fully understand how the key frames are selected, and how many key frames are used in the proposed method. I would appreciate if authors could further clarify that.\n\n-- Are the key frames edited independently and then propagated to non-key frame views, or are the key frames also edited sequentially using the previously edited ones? \n\n-- Is there any ordering when the edited key-frames warped to other views sequentially, or is the key frames are selected randomly and independently?\n\n-- Some of the limitations of the method are discussed in the limitation section. It would have been more helpful to also include some visualizations of the failure cases, especially the ones which do not happen in IN2N.\n\nLIMITATIONS:\nThe limitations and social impact have been addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to edit a NeRF using natural language. The authors achieve this by first “generating” an edited version of the input dataset and then training a NeRF on the edited dataset. To generate the said dataset, the authors train a NeRF on the original dataset for depth, with which they then propagate their keyframe edits (made with Instruct-Pix2Pix) to other views. Because NeRF depth is noisy, the authors use a diffusion-based blending network to refine those images into a cleaner dataset for the next-stage NeRF, which simply learns the radiance field from the new, edited images.\n\nSTRENGTHS:\nThe paper tackles the difficult problem of editing 3D scenes in a multi-view-consistent way. The proposed method is general, being applicable to humans, objects, and even unbounded scenes. It also takes advantage of physics-based operations such as projections and combines them with learning to handle noise and uncertainty (e.g., in NeRF depth). The paper produces plausible results (only up to a low resolution though).\n\nWEAKNESSES:\nThe major weakness, I think, is the complexity of the pipeline that requires the user to train a NeRF for depth, run inference on keyframes with Instruct Pix2pix, project edited frames to other views, run a diffusion model to perform blending, and finally train another NeRF on the edited images. It does not sound like an easy-to-use pipeline that can be flexibly extended or built upon. A complicated pipeline like this has many points of failure, too, as compared with a simpler pipeline.\n\nThe question then becomes which part of the pipeline we may cut to achieve similar results. The authors claim it’s crucial to perform diffusion-based blending to get around the artifacts due to direct mixup. Indeed, the results look quite blurry. Given (the 2nd-stage) NeRF also blurs the results, I think the authors should ablate the blending module, i.e., directly feeding the mixup output into the 2nd-stage NeRF. My hunch is NeRF’s blurring effect may already achieve what the additional diffusion model is doing, To be clear, I do think blurry results are a weakness of this paper, too.\n\nI think one crucial experiment on the number of edited keyframes is missing: how many keyframes does one need to edit to achieve good results? Why not just run Instruct Pix2pix on all views instead of doing mixup and then blending?\n\nQUESTIONS:\nBecause I don’t expect the authors to change their method, I’m primarily interested in hearing about points #2 and #3 from the Weaknesses section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced ViCA-NeRF, a method for 3D-consistent NeRF editing using diffusion models. The proposed method decouples NeRF editing with two steps. In first stage (i.e. dataset editing), key views are sampled and edited via Instruct-Pix2Pix model and propagated to other views using the NeRF depth maps, obtaining a mixup dataset. In second stage (NeRF training), the edited dataset can be directly used for training the NeRF model. The authors proposed  a blending refinement model to enhance the quality that enables consistent NeRF editing results.\n\nSTRENGTHS:\nThe proposed method demonstrated 3D consistent NeRF editing results and often achieves better editing image quality than baseline Instruct-NeRF2NeRF and NeRF-Art. Concretely, here are some strengths of the proposed method:\n- Guided Editing: The method uses key views for consistent and detailed 3D scene manipulation;\n- Depth-guided Correspondence Matching: Allows projecting one edited view onto all others for consistency and detail;\n- Key Frame Sampling and Mixup: This reduces the computational load by representing the entire scene with a few key frames; \n- Blending Refinement Model: Reduces noise in depth information and resolves incongruities between 2D edits and their 3D context;\n- Warm-Up and Post-Refinement Strategy: Enhances editing quality and efficiency;\n- Semantically Meaningful Edits: The method can make significant edits based on textual prompts.\n\nIn summary, the method is capable of performing 3D consistent NeRF editing with text-guided diffusion pipeline. The quality of the results demonstrate improvements over previous approaches, sufficient ablation and comparisons are shown.\n\nWEAKNESSES:\nThere are some limitations and concerns. \n- Firstly, I was wondering what happens if the NeRF depth or geometry is bad, such as on glossy or specular surfaces. In such case, the depth-guided correspondence matching would decrease its accuracy. Considering most scenes shown in the paper are diffuse, it'd be interesting to have one such example as an ablation.\n- Results are rather blurry, such as Flowers in Fig.1, Cyberpunk Style in Fig. 4 and Fireworks in Fig. VI. In contrast, the \"original NeRF\" on the left exhibits pretty sharp image details. What's the source of blurriness --- Is it due to key frame selection, depth quality, or diffusion model? Some analysis and (at least) discussions are necesssary and helpful. \n- How's Equation (2) proposed? And the hyperparameter phi = 0.3. \n- Can local edits be enabled? Currently the pipeline is, similar to Instruct-NeRF2NeRF, heavily based on Instruct-Pix2Pix. This is typically diffusion on the whole image (i.e. background is also changing and being modified). Would be nice to support locally editing of a person without touching the background in some cases.\n\nQUESTIONS:\nSome related works on 3D consistent NeRF editing:\n- PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields. CVPR 2023.\n- Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field. CVPR 2023.  \n- ARF: Artistic Radiance Fields. ECCV 2022. \n- SNeRF: Stylized Neural Implicit Representations for 3D Scenes. SIGGRAPH 2022.\n- NeRFshop: Interactive Editing of Neural Radiance Fields. I3D 2023.\n\nLIMITATIONS:\nDiscussed in weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for editing Neural Radiance Fields using text instructions. Starting from an initial trained NeRF, the method samples a set of key views which are then edited using Instruct-Pix2Pix. Given the depth maps rendered from the NeRF, the method computes correspondences between different views and propagates the edits to the remaining views. To minimize the artifacts from warping the paper proposes to use a blending procedure that involves 2 Instruct-Pix2Pix models.\n\nSTRENGTHS:\n1. The paper identifies a known issue with performing per-frame edits of Neural Radiance Fields (inconsistencies between views) and proposes an intuitive method to address them. This involves selecting key frames and then use the NeRF geometry to propagate changes to the remaining views. The approach overall is technically sound.\n2. Both the qualitative and quantitative results are good, and are on-par with Instruct-Nerf2Nerf. Given that the proposed method is about 3 times faster than Instruct-Nerf2Nerf then it is a valid, more lightweight alternative.\n3. The paper is well-written and easy to follow. The figures are clear and most components of the method are adequately explained in the text.\n\nWEAKNESSES:\n1. The paper fails to show that all components of the method are needed. For example by looking at Figure 6, I cannot seem to see any difference between the \"blended\" and \"post-refined\" images.\n\nQUESTIONS:\n1. It is very hard to tell if the method is significantly better than Instruct-Nerf2Nerf. The quantitative evaluation in the Supplementary shows very similar numbers. The text-image direction score and the temporal consistency score are correlated with the perceptually better results but there is no direct relationship, so it's hard to quantify which method is better. At the same time, it's hard to tell from a handful of qualitative results which method is better. This is not a problem of this paper, it's generally an issue with editing papers where there is no \"correct\" solution. I do not have a good solution for this. \n2. The keyframe selection criterion seems a little bit arbitrary. How does the value of $\\phi$ affect the final result?\n\nLIMITATIONS:\nThe authors have an extended discussion on the limitations of the method which I appreciate. There is also a discussion on the societal impact in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for 3D editing via text instruction that is view consistent. It leverages on the idea from Instruct-NeRF2NeRF, but additionally propagates the edits into multiple views in order to achieve multi-view consistency. To achieve such, a geometry regularization and a learned regularization were used, where the former uses depth for correspondence matching and the latter uses the typical 2D prior as used by Instruct-Pix2Pix but instead uses the average of the latent codes. Experiments show the effectiveness of the proposed method on 10 scenes.\n\nSTRENGTHS:\nThe paper proposes a way to enable 3D editing through text instruction that is multi-view consistent. The motivation of the proposed pipeline is sound and makes sense. It is also simple, which makes it easy to follow. Experiment demos show nice qualitative examples as well as qualitative ablations.\n\nWEAKNESSES:\nThe main concern I have with the paper is the thoroughness of the experiments. Specifically, in the main paper only qualitative results are included, and the scenes and examples presented are limited. The supplementary included a quantitative metric, but this is also only on 10 scenes.  It also mentioned that the pipeline is heavily reliant on depth, which makes one wonder whether the limited scenes shown is because of the failure cases with depth. Including failure cases might be helpful.\n\nQUESTIONS:\nA critical question I had is the details of the 10 training scenes, specifically, how many input views are used for training? The number of input views would give an idea on how critical the edit propagation to the different views is. Moreover, both the edit propagation and the geometric regularizer rely on depth inferred from the NeRF, and the correctness of NeRF depth is highly dependent on the number of training views. The failure case of the original Instruct-NeRF2NeRF is when the 2D diffusion samples is not consistent in 3D, which is also dependent on how many views are there. Perhaps an ablation on this would also be meaningful and make the contribution more convincing.\n\nLIMITATIONS:\nThe authors included a limitation section in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}