# Human Pose Estimation Using MediaPipe Pose and Optimization Method Based on a Humanoid Model
## 1 Introductions
In spite of continuous technological advances, the deep learning methods of 3D pose estimation from 2D images should solve challenging problems, including a lack of in-the-wild datasets, a huge demand for various posture data, depth ambiguities, and a large searching state space for each joint [9]. Furthermore, a high-performance PC equipped with many GPUs is essential for executing deep learning packages. Firstly, collecting a large in-the-wild dataset with 3D annotation is very intensive, and thus building the popular datasets of HumanEva [19] and Human3.6M [20] requires expensive motion capture systems and many subjects and experiments. Secondly, human pose has an infinite number of variants according to camera translation, body orientation, differences in height and body part ratio, etc. Thirdly, the depth ambiguity problem arises because different 3D poses can be mapped to a single 2D pose, which is known to be mitigated using temporal information from a series of images or multi-viewpoint images [21]. Lastly, the requirement for at least 17-dimensional joint space is also a high-order problem to solve with conventional optimization methods, and thus optimization results are time consuming and unacceptable in precision. In this paper, to run a human pose estimation package on an SBC installed in a mobile robot, a new type of two-stage pose estimation method is proposed. The first stage of 2D pose estimation is performed with MediaPipe Pose [22], and the second stage of estimating joint angles is carried out with a fast optimization method, uDEAS [23] based on an elaborate humanoid model. We propose a 3D full-body humanoid model whose reference coordinate frame is located at the center of the pelvis, i.e., root joint, and three DoF (Degree of Freedom) lumbar joints are newly added to the center. The three lumbar joints of twist, flexion, and lateral flexion can make poses in which only the upper body rotates or bends, and thus they are indispensable joints necessary to create various natural poses, such as yoga, sitting, and lying, amongst others. However, some recent deep learning methods lack these core joints when modeling the human body [5; 6]. In addition, the joint rotation polarity rules for all the humanoid joints are designed to be consistent with those of the Vicon motion capture system [24], which bridges numerous physiological research results on various activities of the human body using Vicon data [25]. An innovative method for resolving the inverse kinematics of the humanoid is to use uDEAS to tune 19 unknown pose-relevant variables for each frame of real-time pose estimation with camera-based or video-based images. This allows the humanoid joint angles to fit the 2D humanoid model that is reprojected from the 3D model to the MPP skeleton as closely as possible. The proposed approach does not require the first problem of a significant amount of human pose data, and a full set of optimization variables is constructed for resolving the second pose variation problem. The third problem of depth ambiguity can be addressed by adding deviation of center of mass from the center of the supporting feet as well as appropriate penalty functions concerning an allowed range of joint angle rotation. The fourth problem can be overcome by employing a fast optimization method, such as uDEAS. For the validation of the proposed approach, several ADL poses were attained by simulation and experiment. We generated simulations of human poses using a humanoid model and given joint angles as ground-truth data, and we allowed uDEAS to estimate the true joint angles by taking simulated poses as the input. In order to check for practicality, gymnastics motion and sudden fall motion were filmed with a camera. The 3D pose estimation results and the obtained joint trajectories were acceptable for application to mobile robots that monitor poses. Unfortunately, most state-of-the-art deep learning methods require CUDA-relevant libraries and a GPU hardware, and we could not apply them to our small mobile robot.