URHand: Universal Relightable Hands
Authors: name: Zhaoxi Chen; name: Gyeongsik Moon; name: Kaiwen Guo; name: Chen Cao; name: Stanislav Pidhorskyi; name: Tomas Simon; name: Rohan Joshi; name: Yuan Dong; name: Yichen Xu; name: Bernardo Pires; name: He Wen; name: Lucas Evans; name: Bo Peng; name: Julia Buffalini; name: Autumn Trimble; name: Kevyn McPhail; name: Melissa Schoeller; name: Shoou-I Yu; name: Javier Romero; name: Michael Zollhöfer; name: Yaser Sheikh; name: Ziwei Liu; name: Shunsuke Saito
Published: 2024-01-10 18:59:51

Abstract: Existing photorealistic relightable hand models require extensive
identity-specific observations in different views, poses, and illuminations,
and face challenges in generalizing to natural illuminations and novel
identities. To bridge this gap, we present URHand, the first universal
relightable hand model that generalizes across viewpoints, poses,
illuminations, and identities. Our model allows few-shot personalization using
images captured with a mobile phone, and is ready to be photorealistically
rendered under novel illuminations. To simplify the personalization process
while retaining photorealism, we build a powerful universal relightable prior
based on neural relighting from multi-view images of hands captured in a light
stage with hundreds of identities. The key challenge is scaling the
cross-identity training while maintaining personalized fidelity and sharp
details without compromising generalization under natural illuminations. To
this end, we propose a spatially varying linear lighting model as the neural
renderer that takes physics-inspired shading as input feature. By removing
non-linear activations and bias, our specifically designed lighting model
explicitly keeps the linearity of light transport. This enables single-stage
training from light-stage data while generalizing to real-time rendering under
arbitrary continuous illuminations across diverse identities. In addition, we
introduce the joint learning of a physically based model and our neural
relighting model, which further improves fidelity and generalization. Extensive
experiments show that our approach achieves superior performance over existing
methods in terms of both quality and generalizability. We also demonstrate
quick personalization of URHand from a short phone scan of an unseen identity.