Spherical Based Human Tracking and 3D Pose Estimation for Immersive Entertainment Production
Creators: name: SHERE, MATTHEW JAMES
Publisher: name: University of Surrey
Year: 2021

description: Human tracking and 3D pose estimation are two core activities of computer vision, identifying
description: and following an individual within a scene in the case of the former, and producing a three-dimensional estimate of an individuals body pose and configuration for the latter. This
description: combination of processes cane applied to scenarios such as entertainment production, home
description: health monitoring or sports analysis, where there is a relatively low person count, and
description: a requirement to know more than just the approximate three-dimensional position of a
description: person. However, such systems generally require non-complimentary camera configurations,
description: requiring two separate but overlapping camera rigs to be established.
description: An ideal solution would be to combine our camera configurations, something which can
description: easily be achieved using wide angle, panoramic or 360° cameras. Through careful placement
description: of these cameras, we simultaneously view the entire scene, and also produce multiple views
description: of an individual in order to inform our pose estimate. However, such cameras bring their
description: own representation problems, hampering the performance of existing solutions, or preventing
description: them from operating entirely. Therefore, we explore this facet of the problem, producing
description: tracking and pose estimation solutions that natively function from 360° imagery.
description: To facilitate this, we firstly contribute a tracker and pose estimation system, operating
description: from a pair of horizontally disjoint 360° cameras. We use provided person segmentation
description: masks to create descriptors suitable for use at differing resolutions, while the specific
description: camera configuration allows us to share these descriptors, using these combined with spatial
description: information to track an individual regardless of their distance from either camera. With a
description: person isolated, we then create a joint-wise pose estimate directly from the spherical coordinate
description: space, eliminating the need for either reprojection operations, or intrinsic calibration
description: information to be provided.
description: Our second contribution reconfigures these cameras to a low, vertical baseline configuration.
description: We simultaneously track each individual in the scene using only two-dimensional joint
description: location estimates, exploiting the camera arrangement to assume an Epipolar relationship. A
description: temporally consistent 3D human pose estimate is then constructed, first as a coarse, Principal Component Analysis (PCA) model, then refined in a joint-wise fashion over successive
description: iterations, smoothing out any unrealistic jumps in motion.
description: Having established tracking in a local area, our final contribution moves beyond the confines
description: of a single room, and tracks individuals as they move throughout a scene comprised of
description: multiple rooms or regions. We perform this with no prior knowledge of the scene layout or
description: content, and use only camera extrinsics and person movements to iteratively build tracks for
description: each individual simultaneously, with each stage informing the next.
description: Overall, we demonstrate that 360° imagery presents many advantages that can be utilised
description: or exploited in both human tracking, and in three-dimensional human pose estimation.
description: We enable tracking in a variety of situations where traditional methods are impractical
description: or impossible, and position methods to provide training data for the next generation of
description: multi-camera, 360° capable deep-learning based tracking approaches. We also produce pose
description: estimates that bridge the gap between multi-view systems and monocular systems.