Fast Registration of Photorealistic Avatars for VR Facial Animation
Authors: name: Chaitanya Patel, name: Shaojie Bai, name: Te-Li Wang, name: Jason Saragih, name: Shih-En Wei

Virtual Reality (VR) bares promise of social interactions that can feel more
immersive than other media. Key to this is the ability to accurately animate a
photorealistic avatar of one's likeness while wearing a VR headset. Although
high quality registration of person-specific avatars to headset-mounted camera
(HMC) images is possible in an offline setting, the performance of generic
realtime models are significantly degraded. Online registration is also
challenging due to oblique camera views and differences in modality. In this
work, we first show that the domain gap between the avatar and headset-camera
images is one of the primary sources of difficulty, where a transformer-based
architecture achieves high accuracy on domain-consistent data, but degrades
when the domain-gap is re-introduced. Building on this finding, we develop a
system design that decouples the problem into two parts: 1) an iterative
refinement module that takes in-domain inputs, and 2) a generic avatar-guided
image-to-image style transfer module that is conditioned on current estimation
of expression and head pose. These two modules reinforce each other, as image
style transfer becomes easier when close-to-ground-truth examples are shown,
and better domain-gap removal helps registration. Our system produces
high-quality results efficiently, obviating the need for costly offline
registration to generate personalized labels. We validate the accuracy and
efficiency of our approach through extensive experiments on a commodity
headset, demonstrating significant improvements over direct regression methods
as well as offline registration.