NaRCan: Natural Refined Canonical Image with Integration of Diffusion
  Prior for Video Editing
Authors: name: Ting-Hsuan Chen; name: Jiewen Chan; name: Hau-Shiang Shiu; name: Shih-Han Yen; name: Chang-Han Yeh; name: Yu-Lun Liu
Published: 2024-06-10 17:59:46

Abstract: We propose a video editing framework, NaRCan, which integrates a hybrid
deformation field and diffusion prior to generate high-quality natural
canonical images to represent the input video. Our approach utilizes homography
to model global motion and employs multi-layer perceptrons (MLPs) to capture
local residual deformations, enhancing the model's ability to handle complex
video dynamics. By introducing a diffusion prior from the early stages of
training, our model ensures that the generated images retain a high-quality
natural appearance, making the produced canonical images suitable for various
downstream tasks in video editing, a capability not achieved by current
canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA)
fine-tuning and introduce a noise and diffusion prior update scheduling
technique that accelerates the training process by 14 times. Extensive
experimental results show that our method outperforms existing approaches in
various video editing tasks and produces coherent and high-quality edited video
sequences. See our project page for video results at
https://koi953215.github.io/NaRCan_page/.