Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images
Authors: name: Haruo Fujiwara; name: Yusuke Mukuta; name: Tatsuya Harada
Published: 2024-06-19 09:36:18

Abstract: We propose a simple yet effective pipeline for stylizing a 3D scene,
harnessing the power of 2D image diffusion models. Given a NeRF model
reconstructed from a set of multi-view images, we perform 3D style transfer by
refining the source NeRF model using stylized images generated by a
style-aligned image-to-image diffusion model. Given a target style prompt, we
first generate perceptually similar multi-view images by leveraging a
depth-conditioned diffusion model with an attention-sharing mechanism. Next,
based on the stylized multi-view images, we propose to guide the style transfer
process with the sliced Wasserstein loss based on the feature maps extracted
from a pre-trained CNN model. Our pipeline consists of decoupled steps,
allowing users to test various prompt ideas and preview the stylized 3D result
before proceeding to the NeRF fine-tuning stage. We demonstrate that our method
can transfer diverse artistic styles to real-world 3D scenes with competitive
quality.