Parametric Human Modelling for Shape and Texture Representation from Video
Creators: name: Bridgeman, Lewis
Publisher: name: University of Surrey
Year: 2023

description: The modelling of human geometry and appearance from videos has been a focus of computer
description: vision researchers for decades, driven by an exciting range of possible applications in video
description: games, healthcare and the TV &amp; film industry. Research in this field has advanced on two
description: fronts: bottom-up methods that reconstruct human geometry from raw image data; and top-down
description: approaches that explain the image data with existing models of human shape. There has been an
description: increased focus into model-based approaches over the last 5 years, largely due to the emergence
description: of new statistical models of human shape and pose - notably the SMPL model - and also due
description: to advances in human joint estimation from images. While model-free human reconstruction
description: methods have been mostly limited to constrained environments, model-based methods can
description: exploit priors in the statistical body model to capture the human geometry in more challenging
description: scenarios, including: monocular video, partially occluded images, and multiple people. In this
description: thesis we demonstrate the advantage of parametric human models in representing the shape and
description: texture of humans from video input, by applying them to a range of tasks.<br>
description: <br>
description: The first contribution demonstrates the effectiveness of a parametric human body model as a
description: tool for generating free-viewpoint video renderings of humans in motion. In this chapter we
description: introduce an optimisation framework for aligning the SMPL body model with multi-view video
description: of a human capture in a studio. The model-based approach consistently provides a full-body
description: reconstruction with fine details around the face and hands. Further, the model-based pipeline
description: allows for considerable compression of the reconstructed sequence: the geometry is encoded
description: simply as a set of model parameters; and the model structure provides a temporally consistent
description: texture map layout, allowing for efficient video compression of the human appearance. These
description: benefits allow for efficient and computationally inexpensive playback in a game engine, and in
description: virtual reality.<br>
description: <br>
description: The second contribution is the model-based reconstruction of multiple people in sports. Sports
description: datasets are especially challenging due to multiple interacting players, heavy occlusion, low
description: effective player resolution and poor calibration. To extend our reconstruction pipeline to multiple
description: people, we introduce an algorithm for the association of 2D pose estimations of multiple people
description: between camera views. We also introduce a novel method for the correction of errors that are
description: often associated with 2D pose estimates. Finally, we introduce an algorithm for the tracking of
description: skeletons over time, which is robust to missing detections. We use the associated and temporally
description: tracked pose detections in our model-based reconstruction pipeline to generate model-based
description: reconstructions of multiple players in sports sequences, despite the heavy occlusion and low
description: detail in the original footage.<br>
description: <br>
description: The third contribution is a method for the capture and modelling of dynamic human texture
description: appearance from a minimal set of input cameras. Previous methods to capture dynamic appearance of a human from multi-view video rely on large camera setups, and typically store
description: texture on a per-frame basis. We generate a parametric reconstruction from minimal cameras
description: (as few as 3) to generate partial texture observations. The parametric reconstruction provides a
description: temporally consistent texture map layout, as well as the human pose each frame. The partial
description: texture observations are combined in a learned framework to generate full-body textures with
description: dynamic details given an input pose. Inspired by traditional multi-view texturing algorithms,
description: we adopt a multi-band weighted loss function to train our network, which minimizes texture
description: artefacts.<br>
description: <br>
description: Our final contribution is a novel continuous displacement field representation for the reconstruction of clothed human body shape from a single image. Recent model-free monocular human
description: shape estimation methods struggle with highly varied poses and occlusions, whereas parametric
description: methods are more robust but limited to tight clothing. Our learnt continuous displacement field
description: representation reconstructs detailed shape for humans in challenging poses. We combine local
description: image features with canonical parametric body model coordinates to build a displacement field
description: that models the distance between the underlying parametric model and the true human surface.
description: Our ParamCDF representation is also able to handle the task of inferring detailed human shape
description: from partially occluded images of humans.<br>