Title: DYNAMIC NOVEL VIEW SYNTHESIS FROM UNSYNCHRONIZED VIDEOS USING GLOBAL-LOCAL MOTION CONSISTENCY PRIOR
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Dynamic novel view synthesis, Unsynchronized multi-view videos, Global-local motion consistency

Abstract: Dynamic novel view synthesis (D-NVS) critically depends on hardware-based synchronization. Current approaches that accommodate unsynchronized settings within the widely-used NeRF or GS frameworks often struggle with local minima, particularly in textureless scenes or when multi-view videos exhibit large misalignments. To tackle this issue, we propose a novel global-local motion consistency prior. Our analysis reveals that the motion, produced by the anisotropy of projected global scene flow across different views, is inherently more effective for correcting temporal misalignments compared to the near-isotropic appearance typically leveraged in NeRF or GS. Building on this insight, we develop a global-local motion consistency loss function that evaluates the alignment between predicted scene flow projections and pre-computed optical flows in multi-view videos. Extensive experiments on public datasets demonstrate the versatility of our loss function across various D-NVS architectures (NeRF and GS), achieving a 50%reduction in synchronization errors and a PSNR improvement of up to 4dB, thereby outperforming existing state-of-the-art methods.

TL;DR: We use a novel motion consistency prior to correct camera timing errors in dynamic novel view synthesis, significantly improving quality and accuracy.