HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian
  Splatting
Authors: name: Yuheng Jiang; name: Zhehao Shen; name: Penghao Wang; name: Zhuo Su; name: Yu Hong; name: Yingliang Zhang; name: Jingyi Yu; name: Lan Xu
Published: 2023-12-06 12:36:53

Abstract: We have recently seen tremendous progress in photo-real human modeling and
rendering. Yet, efficiently rendering realistic human performance and
integrating it into the rasterization pipeline remains challenging. In this
paper, we present HiFi4G, an explicit and compact Gaussian-based approach for
high-fidelity human performance rendering from dense footage. Our core
intuition is to marry the 3D Gaussian representation with non-rigid tracking,
achieving a compact and compression-friendly representation. We first propose a
dual-graph mechanism to obtain motion priors, with a coarse deformation graph
for effective initialization and a fine-grained Gaussian graph to enforce
subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with
adaptive spatial-temporal regularizers to effectively balance the non-rigid
prior and Gaussian updating. We also present a companion compression scheme
with residual compensation for immersive experiences on various platforms. It
achieves a substantial compression rate of approximately 25 times, with less
than 2MB of storage per frame. Extensive experiments demonstrate the
effectiveness of our approach, which significantly outperforms existing
approaches in terms of optimization speed, rendering quality, and storage
overhead.