SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiahua Dong Yu-Xiong Wang
{jiahuad2, yxw}@illinois.edu
"Turn him into iron man"
Instruct-NeRF2NeRF
ViCA-NeRF (Ours)
"Add some flowers"
Figure 1: Our ViCA-NeRF is the first work that achieves multi-view consistent 3D editing with text instructions, applicable across a broad range of scenes and instructions. Moreover, ViCA-NeRF exhibits controllability, allowing for early control of final results by editing key views. Notably, ViCA-NeRF is also efficient, surpassing state-of-the-art Instruct-NeRF2NeRF by being 3 times faster.
We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is available at: https://github.com/Dongjiahua/VICA-NeRF.
The recent advancements in 3D reconstruction technology, exemplified by the neural radiance field (NeRF) [1] and its variants, have significantly improved the convenience of collecting real-world
3D data. Such progress has opened up new opportunities for the development of various 3D-based applications. By capturing RGB data from multiple views and obtaining corresponding camera parameters, NeRF enables efficient 3D representation and rendering from any viewpoint. As the availability of diverse 3D data increases, so does the demand for 3D editing capabilities.
However, performing 3D editing on NeRF is not straightforward. The implicit representation of NeRF makes it challenging to directly modify the 3D scene. On the other hand, since NeRF is trained on RGB images, it can effectively leverage a wealth of existing 2D models. Motivated by this, state-of-the-art methods, such as Instruct-NeRF2NeRF [2], apply 2D editing on individual images via a text-instructed diffusion model (e.g., Instruct-Pix2Pix [3]), and then rely exclusively on NeRF to learn from the updated dataset with edited images and propagate the editing information across different views. Despite yielding encouraging results, this indirect 3D editing strategy is time-consuming, because of its iterative process that requires cyclically updating the image dataset while progressively fine-tuning NeRF parameters. And more seriously, due to the lack of inherent 3D structure, the editing results from the 2D diffusion model often exhibit significant variations across different views. This raises a critical concern regarding 3D inconsistency when employing such a strategy, particularly in cases where the editing operation is aggressive, e.g., "Add some flowers" and "Turn him into iron man" as illustrated in Figure 1.
To overcome these limitations, in this work we introduce ViCA-NeRF, a View-Consistency-Aware NeRF editing method. Our key insight is to exploit two additional sources of regularization that explicitly connect the editing status across different views and correspondingly propagate the editing information from edited to unedited views, thus ensuring their multi-view consistency. The first one is geometric regularization. Given that depth information can be obtained from NeRF for free, we leverage it as a guidance to establish image correspondences between different views and then project edited pixels in the edited image to the corresponding pixels in other views, thereby enhancing consistency. This approach allows us to complete preliminary data modifications before fine-tuning NeRF, as well as enabling users to determine 3D content by selecting the editing results from key 2D views. Moreover, as our method allows for direct and consistent edits, it is more efficient by avoiding the need for iterative use of the diffusion model.
The second one is learned regularization. The depth information derived from NeRF can often be noisy, yielding certain incorrect correspondences. To this end, we further align the latent codes in the 2D diffusion model (e.g., Instruct-Pix2Pix) between edited and unedited images via a blending refinement model. Doing so updates the dataset with more view-consistent and homogeneous images, thereby facilitating NeRF optimization.
Our ViCA-NeRF framework then operates in two stages, with these two crucial regularization strategies integrated in the first stage. Here, we edit key views and blend such information into every viewpoint. To achieve this, we employ the diffusion model for blending after a simple projection mixup. Afterward, the edited images can be used directly for training. An efficient warm-up strategy is also proposed to adjust the editing scale. In the second stage, which is the NeRF training phase, we further adjust the dataset. Our method is evaluated on various scenes and text prompts, ranging from faces to outdoor scenes. We also ablate different components in our method and compare them with previous work [2]. Experimental results show that ViCA-NeRF achieves higher levels of consistency and detail, compared with the state of the art.
Our contributions are three-fold. (1) We propose ViCA-NeRF, the first work that explicitly enforces multi-view consistency in 3D editing tasks. (2) We introduce geometric and learned regularization, making 3D editing more flexible and controllable by editing key views. (3) Our method significantly improves the efficiency of 3D editing, achieving a speed 3 times faster than the state of the art.
Text-to-Image Diffusion Models for 2D Editing. Diffusion models have recently demonstrated significant capabilities in text-to-image editing tasks [3-8]. Earlier studies have leveraged pre-trained text-to-image diffusion models for image editing. For instance, SDEdit [8] introduces noise into an input image and subsequently denoises it through a diffusion model. Despite achieving reasonable results, this approach tends to lose some of the original image information. Other studies have concentrated on local inpainting using provided masks [4, 5], enabling the generation of corresponding content within the masked area with text guidance. More recently, Instruct-Pix2Pix [3] proposes an
Figure 2: Overview of our ViCA-NeRF. Our proposed method decouples NeRF editing into two stages. In the first stage, we sample several key views and edit them through Instruct-Pix2Pix. Then, we use the depth map and camera poses to project edited keyframes to other views and obtain a mixup dataset. These images are further refined through our blending model. In the second stage, the edited dataset is directly used to train the NeRF model. Optionally, we can conduct refinement to the dataset according to the updated NeRF.
approach to editing images based on text instructions. Instruct-Pix2Pix accepts original images and text prompts as conditions. Trained on datasets of editing instructions and image descriptions, this approach can effectively capture consistent information from the original image while adhering to detailed instructions.
In this paper, we adopt Instruct-Pix2Pix as our editing model. Importantly, we further explore the potential of Instruct-Pix2Pix by extending its application to editing, blending, and refinement tasks in more challenging 3D editing scenarios.
Implicit 3D Representation. Recently, neural radiance field (NeRF) and its variants have demonstrated significant potential in the field of 3D modeling [1, 9-14]. Unlike explicit 3D representations such as point clouds or voxels, NeRF uses an implicit representation to capture highly detailed scene geometry and appearance. While training a conventional NeRF model can be time-consuming, Instant-NGP [10] accelerates this process by training a multi-layer perceptron (MLP) with multi-resolution hash input encoding. Notably, NeRFStudio [9] offers a comprehensive framework that can handle various types of data. It modularizes each component of NeRF, thereby supporting a broad range of data. These efforts bridge the gap between image data and 3D representation, making it possible to guide 3D generation from 2D models. Our work uses the 'nerfacto' model from NeRFStudio to fit real scenes. We also leverage the depth estimated from the model to maintain consistency.
3D Generation. Inspired by the success of 2D diffusion models [15, 16], 3D generation has drawn great attention in recent years [17-21]. Dream Fusion first proposes to use a score distillation sampling (SDS) loss as the training target. Later, Magic3D [20] generates higher resolution results by a coarse-to-fine procedure. Various scenarios have been explored, including generating a detailed mesh from text prompts [21] and generating novel objects based on images from a few views [18].
Rather than generating new content, our work focuses on editing the 3D representation with text prompts. By explicitly leveraging 2D editing, we provide a more direct approach to control the final 3D result.
NeRF Editing. Because of the implicit representation of NeRF [1], editing NeRF remains a challenge. Early attempts [22-32] perform simple editing on shape and color, by using the guidance from segmentation masks, text prompts, etc. NeuralEditor [33] proposes to exploit a point cloud for editing and obtain the corresponding NeRF representation. Another line of work tries to modify the content through text prompts directly. NeRF-Art [34] uses a pre-trained CLIP [35] model to serve as an additional stylized target for the NeRF model. More recently, Instruct-NeRF2NeRF [2] and Instruct-3D-to-3D [36] leverage Instruct-Pix2Pix [3] for editing. They can edit the content of NeRF with language instructions. However, both require using the diffusion model iteratively in the NeRF training process, making them less efficient. More importantly, though it shows promising results on real-world data, Instruct-NeRF2NeRF can only work when the 2D diffusion model produces consistent results in 3D. Thus, it is unable to generate a detailed result where 2D edits tend to be different, and it also suffers from making diverse results from the same text prompt.
Figure 3: Illustration of mixup procedure and blending model. We first mix up the image with the edited key views. Then, we introduce a blending model to further refine it. The blending model utilizes two modified Instruct-Pix2Pix ('Inp2p') processes. In each process, we generate multiple results and take their average on the latent code to decode a single final result.
In our work, we propose to use depth guidance for detailed 3D editing. Our ViCA-NeRF only needs to update the dataset at the start of training and further refine it once, making it significantly more efficient. In addition, our method results in more detailed and diverse edit results compared with Instruct-NeRF2NeRF. As several key views guide the whole 3D scene, we allow users to personalize the final effect on NeRF by choosing 2D edit results.
Our ViCA-NeRF framework is shown in Figure 2. The main idea is to use key views to guide the editing of other views, ensuring better consistency and detail. Given the source NeRF, we first extract each view's depth from it and sample a set of key views. Then, we edit these key views using Instruct-Pix2Pix [3] and propagate the edit result to other views. To address the problem of incorrect depth estimation from NeRF, we perform mixup on each image and then refine it through our blending model. The obtained data can be used directly for NeRF training. Optionally, we can perform post-refinement to the dataset after obtaining more consistent rendering results from NeRF.
We first elaborate on NeRF, including how we obtain rendered RGB images and coarse depth from the NeRF model. Subsequently, we present Instruct-NeRF2NeRF, a diffusion model for 2D editing. With this model, we can perform 2D editing using text and then further enable 3D editing.
Neural Radiance Field. NeRF [1] and its variants [9, 10] are proposed for 3D scene representation. The approach models a 3D scene as a 5D continuous function, mapping spatial coordinate  $(x,y,z)$  and viewing direction  $(\theta ,\phi)$  to volume density  $\sigma$  and emitted radiance  $c$ . For a given set of 2D images, NeRF is trained to reproduce the observed images. This is achieved by volume rendering, where rays are traced through the scene and colors are accumulated along each ray to form the final rendered image. While NeRF does not explicitly predict depth, one common approach is to compute a weighted average of the distance values along each ray, where the weights are the predicted volume densities. This process can be conceptualized as approximating the depth along the ray.
Instruct-Pix2Pix. Instruct-Pix2Pix [3] is a 2D diffusion model for image editing. It takes a text prompt  $c_{T}$  and a guidance image  $c_{I}$  as conditions. Given an image  $z_{0}$ , it first adds noise to the image to create  $z_{t}$ , where  $t$  is the timestep to start denoising. With denoising U-Net  $\epsilon_{\theta}$ , the predicted noise can be calculated as
$$
\widetilde {\epsilon} = \epsilon_ {\theta} \left(z _ {t}, t, c _ {I}, c _ {T}\right). \tag {1}
$$
When a predicted noise is calculated, Instruct-Pix2Pix follows the standard denoising process to update the image  $z_{t}$  step by step and obtain the final result  $z_{0}$ , which is an edited image consistent with the initial image and text instruction.
"Add some flowers"
"Turn him into a robot"
"Turn the bear into a robot bear"
Original NeRF
Instruct-NeRF2NERF
Figure 4: Qualitative comparison with Instruct-NeRF2NeRF. Our ViCA-NeRF provides more details compared with Instruct-NeRF2NeRF. In addition, our ViCA-NeRF can handle challenging prompts, such as "Turn him into a robot," whereas Instruct-NeRF2NeRF fails in such cases.
"Turn the bear into a grizzly bear"
"Make it surrounded by flowers"
Instruct-NeRF2NERF
3.2 Depth-Guided Sequential Editing
In our approach, we choose to reflect the user's 2D edits in 3D. Thus, we propose a sequential editing strategy. By defining certain key views, we sequentially edit them by projecting the previous edits to the next key view. This is the geometric regularization that explicitly enforces the consistency between different views.
Sequential Editing with Mixup. The overall procedure of editing is simple yet effective, which consists of editing key views and projecting to other views. First, we select a sequence of key views starting from a random view. Then, we sequentially use Instruct-Pix2Pix to edit each view and project the modification to the following views. At the same time, non-key views are also modified by projection. After editing all key views, the entire dataset is edited.
Given that we project several key views onto other views when they are edited, each view could be modified multiple times. Only pixels that have not yet been modified are altered to prevent inadvertent overwriting. Therefore, modifications from key views are mixed up with each view.
Consequently, our method yields a final result that aligns closely with the modified views, facilitating a controllable editing process. Since previously modified pixels are not changed until blending refinement (explained in Section 3.3), the edits of the first few views can dominate the final result.
Key View Selection. We employ key views to make the editing process consistent. This strategy allows the entire scene to be constrained by a handful of edited views. To ensure the edited views maintain consistency and bear similar content, we define the modified ratio  $\rho$  for each view as the fraction of modified pixels. Then, we use the following equation to select the view with the largest
weight  $w$  as the next key view:
$$
w = \left\{ \begin{array}{c c} \rho & \rho <   \phi \\ \phi - (\rho - \phi) & \rho \geq \phi , \end{array} \right. \tag {2}
$$
where  $\phi$  signifies the maximum desired ratio, which is set to  $\phi = 0.3$  in all experiments. Intuitively, each time the next selected key view will have a suitable overlap with the edited region from each of the previous key views. The key view selection ends when all views' minimum  $\rho$  exceeds a certain threshold.
Depth-Guided Projection. Modern NeRF models are capable of generating depth maps directly, thereby making it possible to construct geometric relationships between different views. We illustrate an example of projecting from edited view  $j$  to view  $i$  as below.
Given an RGB image  $I_{i}$  and its corresponding estimated depth map  $D_{i}$ , the 3D point cloud associated with view  $i$  can be calculated as:
$$
p _ {i} = P ^ {- 1} \left(D _ {i}, K _ {i}, \xi_ {i}\right). \tag {3}
$$
Here,  $K_{i}$  denotes the intrinsic parameters,  $\xi_{i}$  represents the extrinsic parameters, and  $P^{-1}$  is a function mapping 2D coordinates to a 3D point cloud.
To find the corresponding pixels on another view, we project this point cloud to view  $j$  as follows:
$$
I _ {i, j} = P \left(I _ {j}, K _ {j}, \xi_ {j}, p _ {i}\right), \tag {4}
$$
where  $I_{i,j}$  signifies the color of the pixels projected from  $I_j$  onto  $I_i$ . In practice, to ensure the correct correspondence and mitigate the effects of occlusion, we compute the reprojection error to filter out invalid pairs. By using this correspondence, we can project pixels from view  $j$  to view  $i$ .
3.3 Blending Refinement Model
The depth information derived from NeRF [1] can often be quite noisy, resulting in noticeable outliers, so only applying the geometric regularization is insufficient. Specifically, while we can filter out incorrect correspondences, the remaining pixels and projections from varying lighting conditions may still introduce artifacts. Furthermore, a reasonable 2D edit in a single view may appear strange when considered within the overall 3D shape. To address these challenges, we introduce a blending refinement model that employs two Instruct-Pix2Pix [3] processes, acting as the learned regularization to improve the overall quality.
Details about the model are shown in Figure 3, where each diffusion process serves unique functions. During the first pass, we use the mixup image as input and the original image as a condition. This procedure aims to purify the noisy mixup image by preserving the structural integrity of the original image. The resulting image appears dissimilar to the consistent mixup image but without any noise. Thus during the second pass, we use the resulting image from the first pass as input and the mixup image as the condition. Given the shared semantic information between these two images, the outcome strives to closely align with the mixup image's detailed characteristics. Doing so allows us to swiftly generate a clean, stylized result, bypassing the need for iterative updates to the dataset during training.
An essential component of our refinement model involves using an average latent code for each of the two processes. We discovered that the diffusion model tends to generate diverse results, making it challenging to closely align with the target structure or content. However, this effect can be substantially mitigated by averaging multiple runs. Specifically, we introduce  $n_r$  noised latent codes and update them independently. Finally, we compute the average of the latent codes and decode it back into the image. This method provides a more stable and consistent output, enhancing the overall quality of the editing process.
3.4 Warm-Up and Post-Refinement Strategies
To make the editing more efficient and further improve the quality, we propose a warm-up strategy along with a post-refinement strategy.
Warm-Up. When editing via instruct-Pix2Pix, the extent of editing changes can be modulated through hyperparameters, such as the guidance scale [15]. However, achieving the desired overall
Figure 5: Comparison on NeRF-Art. We compare the editing results based on NeRF-Art's [34] sequences and edits. Our ViCA-NeRF produces more detailed information and achieves more substantial changes to the content.
style remains a problem. Although Instruct-NeRF2NeRF ensures consistent stylization, it necessitates iterative updates, which could inefficiently involve a large number of iterations.
To overcome these limitations, we introduce a strategy to efficiently warm up the editing process by blending edits directly. This approach is performed before the entire editing process. Specifically, we randomly select a view to edit, and then project the edited view onto all other views. For a given view  $i$  and an edited view  $j$ , an update is calculated as follows:
$$
I _ {i} ^ {\prime} = \lambda I _ {i} \odot m _ {w} + (1 - \lambda) I _ {i, j} \odot m _ {w} + I _ {i} \odot (1 - m _ {w}). \tag {5}
$$
Here,  $m_w$  denotes the binary mask for the projected area, and  $\lambda$  is a hyperparameter employed to control the ratio preserved from the original pixel value. This efficient warm-up strategy accelerates the editing process while preserving a high degree of consistency in the stylization.
Post-Refinement. To further enhance the consistency of the edited results, we employ a post-refinement strategy once the NeRF has been trained on our updated data. We still utilize the modified Instruct-Pix2Pix with averaging architecture to generate a more accurate target for each view. Unlike the first stage, this time we employ the rendered image as input while continuing to use the mixup image as a condition. This post-refinement strategy further refines the consistency and quality of the final output, contributing to a more cohesive and visually appealing result.
We conduct experiments on various scenes and text prompts. All our experiments are based on real scenes with NeRFStudio [9]. We first show some qualitative results and comparisons between our method and Instruct-NeRF2NeRF [2]. For artistic stylization, we test our method with the cases from NeRF-Art [34]. Experiments show that we achieve more detailed edits. Based on these scenes, we further conduct ablation studies on our method, including the effects of different components in the framework, the warm-up strategy, failure cases, and representative hyperparameters. We also conduct quantitative evaluation of the results, testing the textual alignment, consistency, and efficiency of our method.
For a fair comparison, our method employs a configuration similar to that of Instruct-NeRF2NeRF. We utilize the 'nerfacto' model from NeRFStudio and Instruct-Pix2Pix [3] as our 2D editing model. For the diffusion model, we apply distinct hyperparameters for different components. During the editing of key views, the input timestep  $t$  is set within the range [0.5, 0.9]. We employ 10 diffusion steps for this phase. In the blending refinement model, we set  $t$  to 0.6 and  $n_r$  to 5, and use only 3 diffusion steps. The reduced number of steps enables the model to operate more quickly, since encoding and decoding are executed only once. The diffusion model provides additional adjustable
Edited 2D Key View
Rendered Result
"Turn him into Link from Zelda"
Figure 6: Different 2D edits and corresponding 3D edits. Our ViCA-NeRF achieves high correspondences between 2D edits and 3D edits, thus enabling users to control the final edits. On the other hand, our method exhibits high diversity with different random seeds.
Origin image
Figure 7: Image update through our ViCA-NeRF. Our blending model effectively eliminates artifacts and boundaries caused by mixup. The post-refinement step further improves the consistency with the mixup image and the 3D content.
Post-Refined
Improving Consistency