Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric
  Videos
Authors: name: Yuheng Jiang; name: Zhehao Shen; name: Yu Hong; name: Chengcheng Guo; name: Yize Wu; name: Yingliang Zhang; name: Jingyi Yu; name: Lan Xu
Published: 2024-09-12 18:33:13

Abstract: Volumetric video represents a transformative advancement in visual media,
enabling users to freely navigate immersive virtual experiences and narrowing
the gap between digital and real worlds. However, the need for extensive manual
intervention to stabilize mesh sequences and the generation of excessively
large assets in existing workflows impedes broader adoption. In this paper, we
present a novel Gaussian-based approach, dubbed DualGS, for real-time
and high-fidelity playback of complex human performance with excellent
compression ratios. Our key idea in DualGS is to separately represent motion
and appearance using the corresponding skin and joint Gaussians. Such an
explicit disentanglement can significantly reduce motion redundancy and enhance
temporal coherence. We begin by initializing the DualGS and anchoring skin
Gaussians to joint Gaussians at the first frame. Subsequently, we employ a
coarse-to-fine training strategy for frame-by-frame human performance modeling.
It includes a coarse alignment phase for overall motion prediction as well as a
fine-grained optimization for robust tracking and high-fidelity rendering. To
integrate volumetric video seamlessly into VR environments, we efficiently
compress motion using entropy encoding and appearance using codec compression
coupled with a persistent codebook. Our approach achieves a compression ratio
of up to 120 times, only requiring approximately 350KB of storage per frame. We
demonstrate the efficacy of our representation through photo-realistic,
free-view experiences on VR headsets, enabling users to immersively watch
musicians in performance and feel the rhythm of the notes at the performers'
fingertips.