New multi‐view human motion capture framework

IET Image ProcessingVolume 14, Issue 12 p. 2668-2674 Special Section: AI-Powered 3D Vision Free Access New multi-view human motion capture framework Yuan Wang, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorFeiyi Xu, School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorChi-Man Pun, Department of Computer and Information Science, University of Macau, Macau SAR, People's Republic of ChinaSearch for more papers by this authorWenqi Xiao, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorJianhui Nie, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorJian Xiong, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorHao Gao, Corresponding Author tsgaohao@gmail.com orcid.org/0000-0003-0148-3713 Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorFeng Xu, School of Software, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this author Yuan Wang, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorFeiyi Xu, School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorChi-Man Pun, Department of Computer and Information Science, University of Macau, Macau SAR, People's Republic of ChinaSearch for more papers by this authorWenqi Xiao, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorJianhui Nie, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorJian Xiong, Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorHao Gao, Corresponding Author tsgaohao@gmail.com orcid.org/0000-0003-0148-3713 Nanjing University of Posts and Telecommunications, Nanjing, People's Republic of ChinaSearch for more papers by this authorFeng Xu, School of Software, Tsinghua University, Beijing, People's Republic of ChinaSearch for more papers by this author First published: 01 October 2020 https://doi.org/10.1049/iet-ipr.2019.1606 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Estimating human pose and shape without markers is a challenging problem. This study proposes a multiple-view markerless human motion capture framework. Firstly, a multi-view camera system is built for capturing real-time images of moving humans on multiple views. Secondly, by employing the OpenPose method, the authors calculate robust 3D key points from 2D key points of the human body, which are estimated from the multi-view images. And dense 3D point cloud is reconstructed from images. Thirdly, they propose a novel SMPL-based method to represent human motion by fitting the SMPL model to 3D key points and 3D point clouds. In order to achieve a more accurate human pose, a penalty term is utilised to solve the problem of error accumulation in the process of human motion capture. In addition, they present a dense mesh template-based SMPL that can be deformed to point cloud to recover a real human body shape. Finally, they map multi-view colour images onto the human mesh model to acquire rendered mesh. The experimental results show that the proposed method improves the accuracy of human pose and realises the 3D human body model more realistic. 1 Introduction Human motion capture is an important issue in the field of computer vision. At present, markerless human motion capture methods are receiving more and more attention from researchers. Markerless motion capture is an appealing solution for digitally recording the biologic motion information with visual signals. It is one of the interesting topics in the area of computer vision and graphics and plays an important role in many fields, such as culture, military, education, and medical treatment, etc. Currently, there are two major challenges existing in the markerless motion capture: one is to capture the high accuracy motion, the other is to achieve stable motion capture and retargeting results over time. In this paper, we propose a markerless human motion capture framework based on a multi-view system. In the proposed framework, we first estimate the multi-view 2D pose of the moving human body. Then we calculate the corresponding 3D human pose according to the 2D pose, and finally reconstruct the surface of the human body to form the anthropomorphic 3D human motion model. Currently, estimating human pose through deep learning techniques is a hot research direction [1–9]. These tasks can be summarised as estimating 2D or 3D human poses from multiple or single views. In our framework, we employ OpenPose [1] to estimate the 2D human pose from a single image. When the human in the image is occluded or at a special angle of view, OpenPose may get an inaccurate pose or make some mistakes when estimating the 2D key points from a single image. In contrast, when the 2D key points are estimated by multi-view images, the points estimated by different views can be mutually constrained and judged. Then we can eliminate the wrong 2D key points and obtain a more accurate 2D pose. Based on the accurate 2D pose, a robust 3D pose also can be estimated. Therefore, we built a multi-view camera system, which can capture moving human images of multi-view accurately with high speed. Furthermore, we propose a new strategy to track and capture human motion. At first, we generate an anthropomorphic 3D human mesh as a template in first frame, which describes the surface details of the human. Meanwhile, the colour images are mapped onto geometric reconstructions. In the subsequent tracking process, the template is deformed to fit a new human motion. Meanwhile, the template is a parametric model based on Skinned Multi-Person Linear Model (SMPL). The framework of human motion capture gets broad applications in the fields of physical fitness, behaviour analysis, 3D virtual clothing. However, SMPL-based solutions show difficulty in achieving accurate predictions. In the process of capturing human motion, error accumulation appears. This is due to the unconstrained nature of SMPL, which allows unrealistic poses and shapes. In this paper, we add a joint rotation penalty item in the objective function to constrain the excessive rotation of joints during human motion. The experimental results show that the penalty term avoids the unnatural rotation and improves the accuracy of the pose estimation. 2 Related work At present, human motion capture systems can be divided into two categories: sensor/marker-based motion capture systems and markerless motion capture systems. Sensor/marker-based motion capture system is an invasive motion capture [10–12]. Such systems typically require different types of sensors or markers to be installed on objects. In general, sensor/marker-based motion capture systems show three advantages: accuracy, real-time, and robustness. For this reason, such motion capture systems have been widely used, especially in the movies, animation, games and other industries. However, the shortcomings of such systems are also obvious [13]: the capture environment requires special customisation and design; moving objects equipped with sensors or markers are difficult to be captured in some applications. For the inadequacy of the invasive motion capture system, the markerless motion capture system provides a good alternative way. In the past decade, researchers have made great achievements on markerless human motion capture systems. Pose estimation: With the development of deep learning technology in recent years, the work of estimating pose of a human body through a single image attracts more attentions [1–9]. Cao et al. [1] proposed a method PAFs (Part Affinity Fields) to estimate the 2D multi-person body pose in real time from a single image. Compared with the existing methods, our advantage is that the speed of detection is not sensitive to the number of people, and the speed is greatly improved while maintaining the detection accuracy. In [2], Bogo et al. first estimated 2D body pose from a single image and then used the SMPL model to create a 3D body shape mesh, allowing it to estimate the 3D pose from the 2D pose. The authors of [3, 5, 6, 8, 9] proposed different methods for estimating 3D human pose from images based on deep learning. Unlike other articles based on a single image, [6] is based on multi-view images. Sun et al. [7] proposed a reparameterised pose representation of bones instead of joints which can estimate both 2D and 3D poses. Rogez et al. [4] introduced a novel LCR-net++ based on the LCR-net to predict 2D and 3D poses of multiple people simultaneously. Surface reconstruction: After estimating the human pose, the surface reconstruction of a human body is usually continued. Currently, the surface reconstruction typically uses parametric human body model such as SCAPE [14] and SMPL [15]. In comparison, the SMPL model has been widely used in various tasks of human surface reconstruction. One of the tasks is to improve or expand the SMPL model to achieve better surface reconstruction [16–19]. Joo et al. [19] proposed a new parametric model 'Adam', which includes models of the body, face, and hand. It can also express the geometry of hair and clothing. Deep learning technology is increasingly used in a variety of applications and plays an important role in image detection, image classification, and image capture [20–22]. Also due to the development of deep learning technology, there are many studies on obtaining 3D surface models of human body directly from images through deep learning [23–30]. At present, most research is to first estimate the human pose and then reconstruct the surface according to obtained pose. Different from these works, Alp Güler et al. [30] directly obtain the surface of human body from a single image based on deep learning. To estimate human body pose or reconstruct body surface based on deep learning required many high quality training data. For achieving high accuracy of 3D models, researchers have established various human datasets [31–33]. Since the SMPL model usually assumes that the objects wear the least clothing, some research on handling of human clothing in actual data is also underway [34–37], which is one of the most difficult problems. Yu et al. proposed a method for dynamically using skeleton information to constrain the reconstruction of 3D human body models. This paper uses the traditional dynamic-fusion approach as an outer layer and utilises SMPL to model the internal skeleton layer. The internal skeleton layer proposed in the paper can effectively improve the reconstruction of 3D human body model under fast movement. The remaining paper is organised as follows. The data is obtained in Section 3. Section 4 introduces our framework. The results of the experiments are shown in Section 5. Then we conclude our paper in Section 6. 3 Data preparation 3.1 Hardware platform Multi-View Camera Systems: To capture human motion, our paper designs a multi-view system with 20 industrial cameras (Blackfly S cameras) including 10 high-resolution (2448 × 2048) cameras and 10 high-speed (170 FPS) cameras. The two types of cameras are uniformly mounted on each side of a 10-sided bracket with a diameter of 6 m and a height of 1.6 m. The high-resolution cameras provide more details for human body reconstruction and high-speed cameras are utilised to capture high-speed human motion. Some images captured by multiple-view system are shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint Images captured from multiple view camera system To capture raw image data, we use four computers equipped with capture cards, each of which is connected to five cameras via data cables. Meanwhile, each camera is connected to a camera trigger module via an SDI cable, and the module is capable of generating a clock signal to initiate and synchronise the capture of all cameras. Moreover, the software SpinView is used to receive raw image data from five cameras and store it in the solid-state drive (SSD) of each computer. When a trigger signal is detected, the camera captures a frame of images. Finally, the system can capture images on 20 different views at the frame rate of 30. During the image acquisition process, the camera is used to collect images of high-speed moving object. Since the quality of the images affects the accuracy of the human pose and the texture details of the human surface, the exposure settings are adjusted to avoid blurred images. In addition, in order to obtain high-quality images, the external light source needs to be particularly bright and stable. The acquired images will be processed to obtain 3D key points and point clouds, as well as texture maps. After the system is set up, all cameras should be calibrated. In the calibration process, we calibrate the intrinsic and extrinsic parameters of cameras through a standard checkerboard pattern. Then, PhotoScan software is utilised to align all 20 cameras in the global coordinate system by the calibrated parameters. 3.2 3D data acquisition 3D key points: This paper employs the OpenPose [1] detector in each image on different views, which produces 2D key points including 25 key points of body and 21 key points of each hand, as shown in Fig. 2b. OpenPose is an open source library based on convolutional neural networks and supervised learning. It can realise pose estimation of human movements, facial expressions, and even finger movements from images or videos. It is suitable for single and multiple peoples, and shows good robustness. OpenPose is the first deep learning based real-time multi-person 2D pose estimation algorithm of the world, and solutions of improving Multi-Person Pose Estimation based on it have sprung up. Fig. 2Open in figure viewerPowerPoint 3D data acquisition (a) Images on one view, (b) 2D key points detection, (c) Projection of 3D key points on the image, (d) 3D point clouds reconstructed by PhotoScan In general, human body pose estimation from a single image depends on experience, which is vague in challenging scenes. Currently, based on deep learning technology, the human body pose estimation from a single image has achieved excellent results. However, in challenging scene of occlusion, the method still has irreparable defects. The 2D key points estimated by occluded image are inaccurate. Compared to a single image, the multi-view system is robust against occlusion, and allow object to move freely in the scene. Meanwhile, OpenPose may make mistakes when estimating the 2D human key points. For example, it makes inconsistent judgments on the left and right directions of the human body in the images of different views, and some key points estimated in challenging scenes with inter-occlusions or motion blur are inaccurate. In response to these problems, we process the acquired 2D key points, such as removing the wrong points and unifying the direction of the human body in each image. In addition, when calculating 3D key points, we set smaller confidence values for the blurred 2D key points to reduce their impact on the final result. This paper utilises triangulation to obtain 3D key points from multiple-view 2D key points, which uses known camera calibration parameters. Then, we can achieve more constraints of 3D key points from 20 views at the same time. At this point, robust 3D key points have been obtained from images of multi-view. 3D point clouds: The 3D human point clouds are important to restore the body surface and evaluate the reconstruction results. Therefore, we use the commercial software PhotoScan to reconstruct 3D point clouds (see Fig. 2d) from the multi-view images, with associated colour and point normal. 4 Proposed framework In the previous section, multi-view images were acquired, and the data was processed to obtain 3D key points and point clouds, both as inputs. In this work, we expect to obtain the human motion model. In our proposed framework, the human parameters in the previous frame will be utilised as a priori in the next frame to track human motion. In this section, the human motion capture framework proposed in this paper will be described in detail. At first, we estimate SMPL parameters in the first frame by fitting the 3D key points and point clouds. Next, the human mesh model in the first frame needs to be accurately generated as a template for subsequent tracking. In addition, in our framework, the final 3D human model is a parametric model based on the SMPL model. Therefore, the SMPL parameters represent the human motion model. On the basis of this, the rest of the chapter is divided into two parts as follows: firstly, an SMPL-based human mesh template is generated in the first frame; afterwards, the subsequent tracking process is described. 4.1 Template-based SMPL 4.1.1 SMPL model The SMPL model was first proposed by Loper et al. in 2015 [15], which is a parameter-based model. Due to the excellent performance, SMPL model becomes one of the most popular human model of the moment, which contains N = 6980 vertices. The vertex positions of SMPL are adapted according to the shape and pose parameters. The movement of the human body is driven by the transformation of K = 24 joints with three rotational Degrees of Freedom (DoF). The SMPL model can be represented by the following equation: (1)where is the linear blend skinning (LBS) function; shape and pose denote PCA shape components and relative angle-axis rotations of joints respect to their father's node in a defined kinematic tree, respectively; t is the global translation in the coordinate system. is the model in rest pose and calculated by the following equation: (2)where is the default template of the SMPL model; and represent the offsets produced by parameters pose and shape, respectively. 4.1.2 Template generated in first frame In the initial frame, SMPL model is used as a template. Then we fit SMPL model to 3D key points and 3D point cloud to estimate pose parameters and generate a human mesh template. First of all, human pose is estimated by the following equation: (3)and human shape is obtained by the following equation: (4)where represents the SMPL model with shape and pose in rest pose; is the object function defined in (9) for estimating SMPL parameters; (3) and (4) are special cases of (9) for having pose and shape, respectively. The formula will be described in detail in Section 4.2. Then the SMPL model is built with human shape and pose. The SMPL model is sparse, which is difficult to express texture of human body surface. Therefore, we do subdivision surface on SMPL model to get more patches for more mesh details. And a dense mesh is obtained with 110, 210 vertices and 220, 416 patches. Meanwhile, we recalculate the blend weight matrix, representing the impact of each joint's rotation on the vertices, and the pose blend shapes, deviating vertices from the rest template. In order to display rich texture details, the mesh is deformed to real point clouds, using Laplacian deformation. To maintain the geometry of the mesh surface, the vertices of the deformed mesh model are obtained by optimising function (5)where is the Laplacian matrix; is the Laplace coordinate containing local characteristic information of the surface; and are the vertex of the deformed mesh and a constraint point, respectively; m denotes the number of constrain vertices and represents the weight of constraint vertex. Afterwards, the rendered human mesh is acquired through mapping multi-view images onto the deformed human mesh in first frame. Finally, the human mesh is transformed into rest pose as a template: (6)where is the inverse of LBS W(); represents the offsets produced by parameters pose. During the movement of human, the shape parameters of body are constant. If the pose parameters are obtained, the current human mesh can be calculated by the formula: (7) (8)The pipeline of generating template in the first frame is shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Pipeline of generating template based on the images in the first frame 4.2 Tracking human motion An accurate template has been obtained in the first frame, and it is applied to track the human motion of each frame. The current human mesh is achieved by deforming template. In the process of tracking, the proposed method utilises the estimation of previous frame as the prior of the current frame. The model parameters of each frame are estimated by solving the following objective function. 4.2.1 Objective function This paper fits the SMPL model to 3D key points and 3D point clouds (introduced in Section 3.2) in the sequence to capture the human motion. The human parameters can be estimated by minimising an object function as follows: (9)where the model parameter is determined by the three terms , and . , and represent the weights of the three parts in the formula, respectively. We use gradient descent to solve the objective function. Each of the terms in (9) is described as follows. Key point term: The term (as shown in (10)) fits some 3D key point detections, which are in correspondence to mesh model. The correspondence between 3D key points and SMPL joint is shown in Fig. 4. The blue points are the joints of SMPL, and the green 3D key points are calculated from multiple-view 2D detections. Note that the 3D key points on the fingers, feet, and face are in correspondence to the surface vertices of the model. Fig. 4Open in figure viewerPowerPoint Correspondence between 3D key points and the key points of the SMPL model (a) Key points of SMPL model, (b) 3D key points (green) and the key points of the SMPL model (blue) The joint positions are determined by shape in the rest pose, and posed joints are produced by transformation matrices which transform the rest pose with pose . Therefore, human shape and pose parameter can be estimated from joint positions. The formula of key point term is (10)where D is the set of available 3D key points in a particular frame; is the joint in model or point on the surface, which is in correspondence to the 3D key points. ICP term: In the first frame, the term fits point clouds to make the SMPL model under clothes, and an accurate human mesh template is built. After the first frame, the template is used to track the human motion of each frame. Firstly, Iterative Closest Point (ICP) is used to find the closest point in points cloud to mesh surface and the correspondence between mesh and point clouds is established. In the first frame, human is about at rest pose, and the clothes are close to the human body surface. With the movement of people, human clothes will be deformed. In order to reduce the effect of clothes deformation, the mesh is expected to be under the clothes. We compute the point-to-plane distance along the normal direction to optimise pose parameter , as shown in the following equation: (11)where N is the number of vertices in mesh model; is the normal of vertex ; is an indicator function returning 1 when the correspondence point lies inside the mesh. Otherwise, the function returns 0. The is obtained by computing the angle between the mesh surface normal and the vector , where connect the vertex in the mesh and the closest point in the 3D point clouds. Penalty term: To capture human motion in a sequence, error accumulation is a challenging problem to be solved. During the tracking of human motion, wrinkles appear around the surface of the human joint, which are caused by the error of human pose. In general, each joint of the human body rotates only a small angle around its skeleton, and the skeleton connects the joint to its child joint. In order to solve the error accumulation and avoid unnatural joint pose, we propose a joint rotation penalty term as follows: (12)where represents the parent joint of the ith joint; is the position of the ith joint; is the pose of the ith joint with three rotational DoF. The penalty term penalises the rotation around its skeleton to optimise joint rotation angle and favour probable poses. 4.2.2 Tracking shape and pose In order to tracking human motion, the human mesh template is used to perform operations. Then we fit to 3D key points and point clouds for accurate pose. The human pose is obtained by minimising the following equation: (13)where is defined as (9); is calculated in the first frame; the shape is constant; the pose and the global translation t need to be optimised to track human motion. Finally, the human mesh in the current frame is generated by transforming the human mesh template with pose in the following equation: (14) (15)and the pseudo-code of tracking human motion is show in Algorithm 1 (see Fig. 5). Fig. 5Open in figure viewerPowerPoint Algorithm1: Pseudo-code of tracking human motion 5 Experimental results and discussion In this section, we present the human mesh reconstructed by our method in human motions. The experimental data is captured by the multiple-view system, which is introduced in Section 3.1. We use a computer with Intel i7 3.4 GHz processor, 2.2 GHz CPU core and 32 GB RAM for reconstruction. The program compiled with C++ runs on the Ubuntu 16.04. We test the proposed method to capture human motion on sequences that include squats, punching, kicking, turning, and other daily motion. In order to verify the performance of joint penalty term proposed in Section 4.2, we do experiments and the comparison of results is shown in Fig. 6. In capturing human motion, some of input and output data are shown in Fig. 7, including images, SMPL models and human mesh. The human mesh with accurate mapping of the acquired colour data is shown in Fig. 8, which displays the details of the texture. Fig. 6Open in figure viewerPowerPoint Comparison between the method with penalty term and without (a) Tracking without penalty term, (b) Tracking with a penalty term Fig. 7Open in figure viewerPowerPoint Reconstruction results of human motion on sequence (a) Image in one view, (b) Result based SMPL, (c) Result based the dense mesh template Fig. 8Open in figure viewerPowerPoint Reconstruction of human mesh shown in different view with the details of the texture Discussion: As can be seen in Fig. 6a, human joints are distorted during tracking without the penalty term, which are marked with red boxes. In comparison, the SMPL model, calculated with penalty term, demonstrates favour pose in Fig. 6b. As mentioned above, error accumulation arises in the process of capturing human motion, which leads to abnormal joint torsion. The proposed framework is based on SMPL, which is a parametric model. However, the parametric SMPL model is unconstrained, which results in unrealistic poses and shapes. Meanwhile, given the position of 3D key points, the value of human pose, which is the rotation vector with three rotational DoFs, is not unique. Therefore, it is difficult to achieve accurate predictions. Then the accumulation of error accumulates results in joint distortion. Therefore, a penalty term is proposed in Section 4.2 to constrain impossible pose. This term eliminates the unnecessary relative angle-axis rotations of joints rotation angle respect to their father's node in a defined kinematic tree. By the comparison of results in Fig. 6, it is proved that our proposed method is feasible and it shows merits in solving the error accumulation problem of motion tracking. In tracking human motion, the human mesh-based SMPL model is shown in Fig. 7b, which is difficult to express the human appearance. In order to reconstruct an anthropomorphic human mesh, human mesh based on the template is shown in Fig. 7c. Our algorithm reconstructs a human mesh with more surface detail in the first frame. To improve performance of capturing and produce more realism, we deform the template to 3D point clouds for reconstructing human shape, and map colour images onto template. Fig. 8 shows the texture of human mesh on views. Meanwhile, the rendered mesh is transformed into the rest pose as template. And in order to reduce the complexity of calculation, the current human mesh is obtained by warping template. Compared to the result based on SMPL, the result of the human mesh based on template in Fig. 7c shows more details. As shown in Fig. 7, the results of human mesh still have defects, which are caused by the lack of views. Due to the limit of cameras' number, the point clouds reconstructed by PhotoScan from multiple-view images are sparse, which causes the shortcomings of the final results. We should improve the multi-view camera system by increasing more cameras to obtain high quality 3D data. 6 Conclusion In this paper, we set up a multi-view system to collect human body images and propose a framework to capture human motion. In order to demonstrate the performance of our method, we run on the data collected by the designed system and achieve excellent results. Then, it is found that more views of 2D key points detections are conducive to obtain robust 3D skeleton, which should effectively improve the accuracy of human pose and shape estimation. Only utilising 3D joint position, our method tracks human pose and shape with excellent performance. However, human joints are prone to unnatural distortions in tracking human motion. Our experimental results demonstrate that the proposed joint rotation penalty term optimises the joint rotation angle and eliminates the unnatural joint rotation due to error accumulation. In order to express the surface details and realism, we obtain a dense and deformed mesh template in first frame. Then the template is warped to fit the point clouds in capturing human motion, which improves the accuracy of estimating human pose. In the future, we will focus on the reconstruction of the details of the human surface texture. 7 Acknowledgments The authors acknowledge support by the National Nature Science Foundation of China (grant nos. 61931012, 61822111, 61571236, 61802204, 61701258, 61727808, 61671268), Beijing Natural Science Foundation (JQ19015, L182052), the Science and Technology on Space Intelligent Control Laboratory (KGJZDSYS-2018-02, 6142208180302), the Research Committee of University of Macau (MYRG2019-00086-FST, MYRG2018-00035-FST), the Science and Technology Development Fund of Macau SAR under grant no. 041-2017-A1. 8 References 1Cao Z., Simon T., Wei S.E. et al.: ' Realtime multi-person 2d pose estimation using part affinity fields'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 7291– 7299 Crossref Google Scholar 2Bogo F., Kanazawa A., Lassner C. et al.: ' Keep it SMPL: automatic estimation of 3D human pose and shape from a single image'. European Conf. on Computer Vision, Amsterdam, Holland, 2016, pp. 561– 578 Crossref Google Scholar 3Tung H.Y., Tung H.W., Yumer E. et al.: ' Self-supervised learning of motion capture'. Advanced Neural Information Processing Systems, Long Beach, CA, USA., 2017, pp. 5237– 5247 Google Scholar 4Rogez G., Weinzaepfel P., Schmid C.: ' Lcr-net + +: multi-person 2d and 3d pose detection in natural images', IEEE Trans. Pattern Anal. Mach. Intell., 2020, 42, (5), pp. 1146– 1161 PubMed Web of Science®Google Scholar 5Butepage J., Black M. J., Kragic D. et al.: ' Deep representation learning for human motion prediction and classification'. Proc. of the IEEE Conf. On Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 6158– 6166 Crossref Google Scholar 6Rhodin H., Spörri J., Katircioglu I. et al.: ' Learning monocular 3D human pose estimation from multi-view images'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 8437– 8446 Crossref Google Scholar 7Sun X., Shang J., Liang S. et al.: ' Compositional human pose regression'. Proc. of the IEEE Int. Conf. on Computer Vision, Venice, Italy, 2017, pp. 2602– 2611 Crossref Google Scholar 8Tome D., Russell C., Agapito L.: ' Lifting from the deep: convolutional 3d pose estimation from a single image'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 2500– 2509 Crossref Google Scholar 9Zhou X., Huang Q., Sun X. et al.: ' Towards 3d human pose estimation in the wild: a weakly-supervised approach'. In Proc. of the IEEE Int. Conf. on Computer Vision, Venice, Italy, 2017, pp. 398– 407 Crossref Google Scholar 10Menache A.: ' Understanding motion capture for computer animation and video games' ( Morgan Kaufmann, USA., 1999) Google Scholar 11Woltring H.J.: ' New possibilities for human motion studies by real-time light spot position measurement', Biotelemetry, 1974, 1, (3), pp. 132– 146 CAS PubMed Google Scholar 12 Vicon motion systems: www.vicon.com, accessed 30 November 2019 Google Scholar 13Park S.I., Hodgins J.K.: ' Capturing and animating skin deformation in human motion', ACM Trans. Graph., 2006, 25, (3), pp. 881– 889 Crossref Web of Science®Google Scholar 14Anguelov D., Srinivasan P., Koller D. et al.: ' SCAPE: shape completion and animation of people', ACM Trans. Graph., 2005, 24, (3), pp. 408– 416 Crossref Web of Science®Google Scholar 15Loper M., Mahmood N., Romero J. et al.: ' SMPL: a skinned multi-person linear model'. ACM Trans. Graph., 2015, 34, (6), p. 248 Crossref Web of Science®Google Scholar 16Pishchulin L., Wuhrer S., Helten T. et al.: ' Building statistical shape spaces for 3d human modeling', Pattern Recognit., 2017, 67, pp. 276– 286 Crossref Web of Science®Google Scholar 17Kim M., Pons-Moll G., Pujades S. et al.: ' Data-driven physics for human soft tissue animation', ACM Trans. Graph., 2017, 36, (4), p. 54 Web of Science®Google Scholar 18Romero J., Tzionas D., Black M.J.: ' Embodied hands: modeling and capturing hands and bodies together', ACM Trans. Graph., 2017, 36, (6), p. 245 Crossref Web of Science®Google Scholar 19Joo H., Simon T., Sheikh Y.: ' Total capture: a 3d deformation model for tracking faces, hands, and bodies'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 8320– 8329 Crossref Google Scholar 20Lu H.M., Li Y.J., Mu S. et al.: ' Motor anomaly detection for unmanned aerial vehicles using reinforcement learning', IEEE Internet Things J., 2018, 5, (4), pp. 2315– 2322 Crossref Web of Science®Google Scholar 21Lu H.M., Li Y.J., Chen M. et al.: ' Brain intelligence: go beyond artificial intelligence', Mobile Netw. Appl., 2018, 23, pp. 368– 375 Crossref Web of Science®Google Scholar 22Lu H.M., Li Y.J., Uemura T. et al.: ' Low illumination underwater light field images reconstruction using deep convolutional neural networks', Future Gener. Comput. Syst., 2018, 82, pp. 142– 148 Crossref Web of Science®Google Scholar 23Kanazawa A., Black M. J., Jacobs D. et al.: ' End-to-end recovery of human shape and pose'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 7122– 7131 Crossref Google Scholar 24Kanazawa A., Zhang J. Y., Felsen P. et al.: ' Learning 3d human dynamics from video'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Los Angeles, CA, USA., 2019, pp. 5614– 5623 Crossref Google Scholar 25Habermann M., Xu W., Zollhoefer M. et al.: ' Livecap: real-time human performance capture from monocular video', ACM Trans. Graph., 2019, 38, (2), p. 14 Crossref Web of Science®Google Scholar 26Alldieck T., Magnor M., Xu W. et al.: ' Video based reconstruction of 3d people models'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 8387– 8397 Crossref Google Scholar 27Varol G., Ceylan D., Russell B. et al.: ' Bodynet: volumetric inference of 3d human body shapes'. Proc. of the European Conf. on Computer Vision, Munich, Germany, 2018, pp. 20– 36 Crossref Google Scholar 28Pavlakos G., Zhu L., Zhou X. et al.: ' Learning to estimate 3D human pose and shape from a single color image'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 459– 468 Crossref Google Scholar 29Shysheya A., Zakharov E., Aliev K.A. et al.: ' Textured neural avatars'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Los Angeles, CA, USA., 2019, pp. 2387– 2397 Crossref Google Scholar 30Alp Güler R., Neverova N., Kokkinos I.: ' Densepose: dense human pose estimation in the wild'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 7297– 7306 Google Scholar 31Bogo F., Romero J., Pons-Moll G. et al.: ' Dynamic FAUST: registering human bodies in motion'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 6233– 6242 Crossref Google Scholar 32Lassner C., Romero J., Kiefel M. et al.: ' Unite the people: closing the loop between 3d and 2d human representations'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 6050– 6059 Crossref Google Scholar 33Varol G., Romero J., Martin X. et al.: ' Learning from synthetic humans'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 109– 117 Crossref Google Scholar 34Pons-Moll G., Pujades S., Hu S. et al.: ' Clothcap: seamless 4D clothing capture and retargeting', ACM Trans. Graph., 2017, 36, (4), p. 73 Crossref Web of Science®Google Scholar 35Lassner C., Pons-Moll G., Gehler P.V.: ' A generative model of people in clothing'. Proc. of the IEEE Int. Conf. on Computer Vision, Venice, Italy, 2017, pp. 853– 862 Crossref Google Scholar 36Yu T., Zheng Z., Guo K. et al.: ' Doublefusion: real-time capture of human performances with inner body shapes from a single depth sensor'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 7287– 7296 Crossref Google Scholar 37Zhang C., Pujades S., Black M.J. et al.: ' Detailed, accurate, human shape estimation from clothed 3D scan sequences'. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, HI, USA., 2017, pp. 4191– 4200 Crossref Google Scholar Volume14, Issue12October 2020Pages 2668-2674 FiguresReferencesRelatedInformation