Humans in 4D: Reconstructing and Tracking Humans with Transformers
Authors: name: Shubham Goel; name: Georgios Pavlakos; name: Jathushan Rajasegaran; name: Angjoo Kanazawa; name: Jitendra Malik
Published: 2023-05-31 17:59:52

Abstract: We present an approach to reconstruct humans and track them over time. At the
core of our approach, we propose a fully "transformerized" version of a network
for human mesh recovery. This network, HMR 2.0, advances the state of the art
and shows the capability to analyze unusual poses that have in the past been
difficult to reconstruct from single images. To analyze video, we use 3D
reconstructions from HMR 2.0 as input to a tracking system that operates in 3D.
This enables us to deal with multiple people and maintain identities through
occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art
results for tracking people from monocular video. Furthermore, we demonstrate
the effectiveness of HMR 2.0 on the downstream task of action recognition,
achieving significant improvements over previous pose-based action recognition
approaches. Our code and models are available on the project website:
https://shubham-goel.github.io/4dhumans/.