WHAC: World-grounded Humans and Cameras
Authors: name: Wanqi Yin; name: Zhongang Cai; name: Ruisi Wang; name: Fanzhou Wang; name: Chen Wei; name: Haiyi Mei; name: Weiye Xiao; name: Zhitao Yang; name: Qingping Sun; name: Atsushi Yamashita; name: Ziwei Liu; name: Lei Yang
Published: 2024-03-19 17:58:02

Abstract: Estimating human and camera trajectories with accurate scale in the world
coordinate system from a monocular video is a highly desirable yet challenging
and ill-posed problem. In this study, we aim to recover expressive parametric
human models (i.e., SMPL-X) and corresponding camera poses jointly, by
leveraging the synergy between three critical players: the world, the human,
and the camera. Our approach is founded on two key observations. Firstly,
camera-frame SMPL-X estimation methods readily recover absolute human depth.
Secondly, human motions inherently provide absolute spatial cues. By
integrating these insights, we introduce a novel framework, referred to as
WHAC, to facilitate world-grounded expressive human pose and shape estimation
(EHPS) alongside camera pose estimation, without relying on traditional
optimization techniques. Additionally, we present a new synthetic dataset,
WHAC-A-Mole, which includes accurately annotated humans and cameras, and
features diverse interactive human motions as well as realistic camera
trajectories. Extensive experiments on both standard and newly established
benchmarks highlight the superiority and efficacy of our framework. We will
make the code and dataset publicly available.