Real-time Monocular Full-body Capture in World Space via Sequential
  Proxy-to-Motion Learning
Authors: name: Yuxiang Zhang; name: Hongwen Zhang; name: Liangxiao Hu; name: Hongwei Yi; name: Shengping Zhang; name: Yebin Liu
Published: 2023-07-03 17:59:45

Abstract: Learning-based approaches to monocular motion capture have recently shown
promising results by learning to regress in a data-driven manner. However, due
to the challenges in data collection and network designs, it remains
challenging for existing solutions to achieve real-time full-body capture while
being accurate in world space. In this work, we contribute a sequential
proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton
sequences and 3D rotational motions in world space. Such proxy data enables us
to build a learning-based network with accurate full-body supervision while
also mitigating the generalization issues. For more accurate and physically
plausible predictions, a contact-aware neural motion descent module is proposed
in our network so that it can be aware of foot-ground contact and motion
misalignment with the proxy observations. Additionally, we share the body-hand
context information in our network for more compatible wrist poses recovery
with the full-body model. With the proposed learning-based solution, we
demonstrate the first real-time monocular full-body capture system with
plausible foot-ground contact in world space. More video results can be found
at our project page: https://liuyebin.com/proxycap.