SlowGuess/NeurIPS_2023_OCR huggingface dataset:
hyperparameters, such as the image guidance scale  $S_{I}$  and the text guidance scale  $S_{T}$ . We adopt the default values of  $S_{I} = 1.5$  and  $S_{T} = 7.5$  from the model without manual adjustment.
For training the NeRF, we use an L1 and LPIPS (Learned Perceptual Image Patch Similarity) loss throughout the process. Initially, we pre-train the model for 30,000 iterations following the 'nerfacto' configuration. Subsequently, we continue to train the model using our proposed method. The optional post-refinement process occurs at 35,000 iterations. We obtain the final results at 40,000 iterations.
4.2 Qualitative Evaluation
Our qualitative results, illustrated in Figure 4 and Figure 5, demonstrate the effectiveness of our method in editing scenes through multiple views. We present results on three scenes with varying complexity, from human figures to large outdoor scenes. We also evaluate the stylization capabilities of NeRF-Art [34] and compare our results with the previous state of the art.
For comparison with Instruct-NeRF2NeRF, we replicate some cases from their work and assess several more challenging cases. For simpler tasks such as "turn the bear into a grizzly bear" or "make it look like it has snowed," our method achieves results similar to those of Instruct-NeRF2NeRF. This may be due to the inherent consistency of diffusion models in handling such scenarios. For more complex tasks that require additional detail, like transforming an object into a robot or adding flowers, our method shows significant improvements. A notable example is the "Spiderman cas," where Instruct-NeRF2NeRF struggles with differentiating lines on the mask, while our method renders a clear appearance.
When compared on NeRF-Art, our method clearly outperforms the previous techniques by exhibiting more details and adhering more closely to the desired style. Although we use the same 2D editing model as Instruct-NeRF2NeRF, we are able to produce more vivid colors and details, which are not achievable with iterative updates.
Moreover, a significant advantage of our method is that the final 3D edits can be directed through several key views. This allows us to achieve consistent edits not only across different 3D views but also between the 2D edits and the 3D renders, as shown in Figure 6.
In our ablation study, we first evaluate the individual effects of mixup, blending, and post-refinement on the quality of editing. Further investigation includes the impact of warm-up iterations and NeRF
Origin Image
warr
Figure 8: Edit distribution with different warm-up configurations. When we use warm-up, the edit distribution becomes closer to the target distribution.
$\mathfrak{u}\mathfrak{p} = 0$
warmup=10
depth estimation accuracy. We also explore the influence of the parameter  $\phi$  on key view selection for efficient editing. Each of these components is crucial for understanding the nuances of our method's performance.
Model Component Ablation. As shown in Figure 7, the mixup image may initially appear very noisy and contain artifacts. When we apply our blending model to refine it, the boundary from copy-pasting and artifacts is removed. However, while maintaining almost all the original content, the appearance of the image still changes slightly. Finally, when we apply our post-refinement strategy to update it, the result improves and becomes similar to the rendered result. Notice that the result after blending is sufficiently good, making further refinement an optional step.
Warm-Up Iteration Ablation. Considering that Instruct-Pix2Pix relies heavily on the input image, modifications to the input can significantly affect the output distribution. Therefore, to validate our warm-up procedure, which is designed to alter the scale of editing, we adjust this parameter from 0 (no warm-up) to 30. As Figure 8 illustrates, our findings indicate that while preserving the primary geometry, our warm-up procedure can effectively act as a controller for adjusting the scale of editing, introducing only minor computational costs.
Ablation for Depth Estimation Errors. The projection procedure is critical in our framework; therefore, inferior depth estimation may affect the final result. We conduct an ablation study focusing on two prevalent scenarios: glossy surfaces and sparser views. As Figure 9a demonstrates, our model can accurately make modifications to the bookshelf. If the reprojection check is omitted, more extensive changes are possible, albeit at the cost of losing original details. With sparser views, as depicted in Figure 9b, our model is capable of editing provided that the NeRF is successfully trained.
Ablation on Key Views. We perform experiments based on the hyperparameter  $\phi$ . Specifically, we test the results with  $\phi$  set to 0, 0.3, and 1, as illustrated in Figure 10. When  $\phi = 0$ , the view with the least overlap is selected, resulting in a clear boundary (highlighted in the red box). In other cases, this issue is mitigated. However, when  $\phi = 1$ , nearly all views are designated as key views, which leads to reduced efficiency.
4.4 Local Editing
Existing methods often induce simultaneous changes in both background and foreground during editing, which contradicts specific user intentions that typically focus on altering distinct objects.
(a) Results with glossy surface
Figure 9: Ablation study with incorrect depth. We study both glossy surfaces and sparser views, where the results demonstrate that as long as NeRF is built successfully, our model can manage incorrect depth estimation.
(b) Ablation study on input views
"Turn him into spider man"
Figure 10: Ablation study on  $\phi$  in keyframe selection. Our keyframe selection successfully avoids the boundary problem and achieves efficiency.
Figure 11: Application for local editing. Our ViCA-NeRF can be applied with local editing. The edits can be done for the targeted object without changing the background (e.g., wall).
To address this discrepancy, we introduce a straightforward, user-friendly local editing technique within our framework. Initially, we employ Segment Anything Thing (SAM) [37] to derive masks encompassing all components within each image. Subsequently, users pinpoint the target instance with a single point to isolate it. Following this, we utilize the depth information to project the identified mask across different views progressively. This process continues as long as the projected mask sufficiently overlaps with the corresponding mask in the current view. Then, we integrate the wrapped segmentation mask with the segmented masks in the current view.
Figure 11 exemplifies the process, resulting in a multi-view consistent segmentation that permits exclusive modification of the human figure, leaving the background intact.
4.5 Discussion
For quantitative evaluation, we adopt the directional score metric utilized in Instruct-Pix2Pix [3] and the temporal consistency loss employed in Instruct-NeRF2NeRF [2]. Due to the undisclosed split and text prompts from Instruct-NeRF2NeRF, we establish various difficulty levels to assess performance. Results and comprehensive details are provided in the supplementary material.
A further benefit of our approach is the decoupling of the NeRF training from the dataset update process. This separation significantly reduces the computational burden typically imposed by the diffusion model at each iteration. The only computational expense arises from the warm-up, blending, and post-refinement stages. We analyze the time cost of the "Face" scene and compare it with Instruct-NeRF2NeRF on an RTX 4090 GPU. Both methods undergo 10,000 iterations; however, Instruct-NeRF2NeRF requires approximately 45 minutes, whereas our ViCA-NeRF only needs 15 minutes.
Limitations. Our method's efficacy is contingent upon the depth map accuracy derived from NeRF, which underscores our reliance on the quality of NeRF-generated results. Although our blending refinement can mitigate the impact of incorrect correspondences to some extent, the quality of the results deteriorates with inaccurate depth information. We also note that akin to Instruct-NeRF2NeRF, the edited outcomes tend to exhibit increased blurriness relative to the original NeRF. We investigate this phenomenon further and provide a detailed analysis in the supplementary material.
In this paper, we have proposed ViCA-NeRF, a view-consistency-aware 3D editing framework for text-guided NeRF editing. Given a text instruction, we can edit the NeRF with high efficiency. In addition to simple tasks like human stylization and weather changes, we support context-related operations such as "add some flowers" and editing highly detailed textures. Our method outperforms several baselines on a wide spectrum of scenes and text prompts. In the future, we will continue to improve the controllability and realism of 3D editing.
This work was supported in part by NSF Grant #2106825, NIFA Award #2020-67021-32799, the Illinois-Inspaper Partnership, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, the IBM-Illinois Discovery Accelerator Institute, and the Amazon Research Award. This work used NVIDIA GPUs at NCSA Delta through allocations CIS220014 and CIS230012 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.
[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021. 1, 3, 4, 6
[2] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-NeRF2NeRF: Editing 3D scenes with instructions. In ICCV, 2023. 2, 3, 7, 10
[3] Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023. 2, 3, 4, 6, 7, 10
[4] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022. 2
[5] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022. 2
[6] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imag: Text-based real image editing with diffusion models. In CVPR, 2023.
[7] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023.
[8] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 2
[9] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In SIGGRAPH, 2023. 3, 4, 7
[10] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1-102:15, 2022. 3, 4
[11] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2020.
[12] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[13] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[14] Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, and Noah Snavely. Neural 3D reconstruction in the wild. In SIGGRAPH, 2022. 3
[15] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 3, 6
[16] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 3
[17] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. *Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation*. In ICCV, 2023. 3