Instant Facial Gaussians Translator for Relightable and Interactable
  Facial Rendering
Authors: name: Dafei Qin; name: Hongyang Lin; name: Qixuan Zhang; name: Kaichun Qiao; name: Longwen Zhang; name: Zijun Zhao; name: Jun Saito; name: Jingyi Yu; name: Lan Xu; name: Taku Komura
Published: 2024-09-11 17:40:21

Abstract: We propose GauFace, a novel Gaussian Splatting representation, tailored for
efficient animation and rendering of physically-based facial assets. Leveraging
strong geometric priors and constrained optimization, GauFace ensures a neat
and structured Gaussian representation, delivering high fidelity and real-time
facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform.
  Then, we introduce TransGS, a diffusion transformer that instantly translates
physically-based facial assets into the corresponding GauFace representations.
Specifically, we adopt a patch-based pipeline to handle the vast number of
Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme
with UV positional encoding to ensure the throughput and rendering quality of
GauFace assets generated by our TransGS. Once trained, TransGS can instantly
translate facial assets with lighting conditions to GauFace representation,
With the rich conditioning modalities, it also enables editing and animation
capabilities reminiscent of traditional CG pipelines.
  We conduct extensive evaluations and user studies, compared to traditional
offline and online renderers, as well as recent neural rendering methods, which
demonstrate the superior performance of our approach for facial asset
rendering. We also showcase diverse immersive applications of facial assets
using our TransGS approach and GauFace representation, across various platforms
like PCs, phones and even VR headsets.