Fast View Synthesis of Casual Videos
Authors: name: Yao-Chih Lee; name: Zhoutong Zhang; name: Kevin Blackburn-Matzen; name: Simon Niklaus; name: Jianming Zhang; name: Jia-Bin Huang; name: Feng Liu
Published: 2023-12-04 18:55:48

Abstract: Novel view synthesis from an in-the-wild video is difficult due to challenges
like scene dynamics and lack of parallax. While existing methods have shown
promising results with implicit neural radiance fields, they are slow to train
and render. This paper revisits explicit video representations to synthesize
high-quality novel views from a monocular video efficiently. We treat static
and dynamic video content separately. Specifically, we build a global static
scene model using an extended plane-based scene representation to synthesize
temporally coherent novel video. Our plane-based scene representation is
augmented with spherical harmonics and displacement maps to capture
view-dependent effects and model non-planar complex surface geometry. We opt to
represent the dynamic content as per-frame point clouds for efficiency. While
such representations are inconsistency-prone, minor temporal inconsistencies
are perceptually masked due to motion. We develop a method to quickly estimate
such a hybrid video representation and render novel views in real time. Our
experiments show that our method can render high-quality novel views from an
in-the-wild video with comparable quality to state-of-the-art methods while
being 100x faster in training and enabling real-time rendering.