SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes
Authors: name: Cheng-De Fan; name: Chen-Wei Chang; name: Yi-Ruei Liu; name: Jie-Ying Lee; name: Jiun-Long Huang; name: Yu-Chee Tseng; name: Yu-Lun Liu
Published: 2024-10-22 17:59:56

Abstract: We present SpectroMotion, a novel approach that combines 3D Gaussian
Splatting (3DGS) with physically-based rendering (PBR) and deformation fields
to reconstruct dynamic specular scenes. Previous methods extending 3DGS to
model dynamic scenes have struggled to accurately represent specular surfaces.
Our method addresses this limitation by introducing a residual correction
technique for accurate surface normal computation during deformation,
complemented by a deformable environment map that adapts to time-varying
lighting conditions. We implement a coarse-to-fine training strategy that
significantly enhances both scene geometry and specular color prediction. We
demonstrate that our model outperforms prior methods for view synthesis of
scenes containing dynamic specular objects and that it is the only existing
3DGS method capable of synthesizing photorealistic real-world dynamic specular
scenes, outperforming state-of-the-art methods in rendering complex, dynamic,
and specular scenes.