# Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar
## 4 Pose Regression Results  
In VR systems, our method obtains input data of the head and two controllers of the user from the physical world, estimates the upper body pose, and visualizes the corresponding pose of the avatar in VR. We implement our algorithm in Python using TensorFlow to evaluate our neural network. The neural network model requires 22 MB of memory. We speed up the neural network operation using a GPU (NVIDIA GeForce GTX1080) with an HTC VIVE VR device and Unity version 2019.3.11f1. The resulting upper body pose estimation of the avatar is depicted in Figure 6. The left image is the self-avatar estimated using our method in VR, and the right image is the user in the physical world. As users move their heads and controllers, the self-avatar follows the users' motion. Our proposed method can successfully generate the upper body of the avatar in a consumer VR system with reasonable accuracy in a short time from the HMD and controllers, maintaining temporal consistency between frames. Since [PERSON] et al. [4] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Joints** & **MPJPE of Training Data (cm)** & **MPJPE of Test Data (cm)** \\ \hline Our method & 2.4026 & 1.7301 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Mean per joint position error (MPJPE) values for our method. previously confirmed that using an IK system with a fair price achieves results comparable to a full motion capture system, we do not compare our model with a motion capture method. To compare our pose regression results with IK results, we test several poses of a user and estimate the upper body poses using both the IK and our methods. We use the Final IK [42] tool from RootMotion in Unity to estimate the upper body pose using an IK method. Figure 7 illustrates the example comparison of our results and IK results for interaction motions such as left-hand blocking in Figure 7a, right-hand blocking in Figure 7b, swaying in Figure 7c, lifting in Figure 7d, right-hand in front in Figure 7e, and folding arms motion in Figure 7f. The red lines are IK results, and the blue lines are our results from the users in the physical world. Figure 8 displays the avatar pose estimation results in VR using two methods; our results in the middle row and IK in the bottom row as the VR user in the top row grabs and throws a ball in the physical world. As depicted in Figure 8, both our pose-estimated avatar and the IK-based avatar follow the motion of the VR user similarly. The IK-based avatar throws a ball naturally, and our pose-estimated avatar also throws a ball naturally. The MSE of our method is about 0.041 cm, and that of the IK method is about 0.042 cm for 6736 frames while playing the standalone VR applications. We also compare the performance of our regression method and that of IK in sample VR applications. We measure the computation times per frame while users play sample VR games and analyze the minimum, maximum, and average computation times. We use the Unity Profiler, a CPU usage profiler module that tracks the time spent on the application's main thread per frame. For computation time calculation, we accumulate time spent on rendering, scripts, physics, and animation categories. We conduct these experiments with standalone and multi-user VR applications over the network. Table 5 for a standalone VR application, presents the computation times of our method and the IK method. The computation time to render a self-avatar using our method is between 3.19 and 14.75 milliseconds for 6736 frames (70 s). As in Table 5, the average computation time to generate a frame is 6.73 milliseconds using our method and 10.23 milliseconds using the IK method. Moreover, our method's maximum computation time is  
Figure 6: Comparison of the virtual reality(VR) user (**right**) and the corresponding self-avatar using our method (**left**). almost five milliseconds faster than IK. Consequently, our approach is generally faster with lower latency than the IK method.