Relightable Full-Body Gaussian Codec Avatars
Authors: name: Shaofei Wang, name: Tomas Simon, name: Igor Santesteban, name: Timur Bagautdinov, name: Junxuan Li, name: Vasu Agrawal, name: Fabian Prada, name: Shoou-I Yu, name: Pace Nalbone, name: Matt Gramlich, name: Roman Lubachersky, name: Chenglei Wu, name: Javier Romero, name: Jason Saragih, name: Michael Zollhoefer, name: Andreas Geiger, name: Siyu Tang, name: Shunsuke Saito

We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for
modeling relightable full-body avatars with fine-grained details including face
and hands. The unique challenge for relighting full-body avatars lies in the
large deformations caused by body articulation and the resulting impact on
appearance caused by light transport. Changes in body pose can dramatically
change the orientation of body surfaces with respect to lights, resulting in
both local appearance changes due to changes in local light transport
functions, as well as non-local changes due to occlusion between body parts. To
address this, we decompose the light transport into local and non-local
effects. Local appearance changes are modeled using learnable zonal harmonics
for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are
highly efficient to rotate under articulation. This allows us to learn diffuse
radiance transfer in a local coordinate frame, which disentangles the local
radiance transfer from the articulation of the body. To account for non-local
appearance changes, we introduce a shadow network that predicts shadows given
precomputed incoming irradiance on a base mesh. This facilitates the learning
of non-local shadowing between the body parts. Finally, we use a deferred
shading approach to model specular radiance transfer and better capture
reflections and highlights such as eye glints. We demonstrate that our approach
successfully models both the local and non-local light transport required for
relightable full-body avatars, with a superior generalization ability under
novel illumination conditions and unseen poses.