# Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar
## 3 Pose Regression Methods
### Pose Regression Model  
The entire process of a pose regression model is shown in Figure 3. Through comparison, we find CNN outperforms DNN in our problem. The hierarchical structures of  
Figure 2: Upper body joints \(j_{1},j_{2},j_{3},j_{4},j_{5},j_{6},j_{7},j_{8}\), and \(j_{9}\) indicate the head, left hand, right hand, left elbow, right elbow, left shoulder, right shoulder, spine, and pelvis. features in CNNs contribute the superior performance. Of paramount importance, our input data comprises two distinct types: position and orientation. We maintain minimal connectivity between these two data types to ensure optimal performance. The correlation between these two different data types is not relevant to our objective of inferring the position of the joints. Separately, treating the position and orientation data is optimal rather than trying to find a correlation between position and orientation values. In light of this, we leverage and adapt an existing CNN architecture to derive separate feature maps from the positions and orientations. We modify the existing Conv2D layer to incorporate the position and orientation data in different depths of our input matrix. Convolutional layers generate feature maps that separately have the position and orientation information. Without this consideration, one datatype could dominate the other after convolution, resulting in the loss of one of the features. Since our architecture has the position and orientation data in different depths, our convolutional layer keeps the position and orientation features rather than one fading away. Position and orientation data in our architecture works similarly to the red, green, and blue channels in the conventional Conv2D layer for image-related problems. The input matrix components are as depicted in Equation (1). Given the 18 input values, which are the position and orientation values of \(j_{1},j_{2},\) and \(j_{3}\), our pose regression model estimates the position values for the target joints \(j_{4},j_{5},j_{6},j_{7},j_{8}\), and \(j_{9}\). In the input matrices, the width is the joint index of the head, left-hand, and right-hand joints, the height is the values along \(x\)-, \(y\)-, and \(z\)-axes, respectively, and the depth is the position and orientation data. \[\left[ \begin{array}{ccc}\text{P}\text{x}_{1}&\text{P}\text{x}_{2}&\text{P} \text{x}_{3}\\ \text{P}\text{y}_{1}&\text{P}\text{y}_{2}&\text{P}\text{y}_{3}\\ \text{P}\text{z}_{1}&\text{P}\text{z}_{2}&\text{P}\text{z}_{3}\\ \end{array} \right] \tag{1}\]  
The structure of our convolutional layers includes a convolution layer with a ReLU activation and two dropout layers, followed by two fully connected layers. The convolutional layer has a filter size of \(3\times 3\times 2\) along the width and height; the stride in the horizontal and vertical directions is one. Our convolution is applied to the cells, and the same weights are applied to the position data in the first depth and orientation data in the second depth. This allows for treating position and orientation as a group and deriving features between joints. A zero-padding solution controls the shrinkage of the output dimension from each convolution layer, and we add zero-padding on the borders. The convolutional  
Figure 3: Our pose regression model. layer derives the features from the input matrices. The first convolutional layer operates on the input matrix to perceive the corresponding features within each frame. The next convolutional layer detects the relevant features from the previous layer's feature map. After a convolution operation, the obtained feature map is equivalent to the corresponding feature detection results. After each convolutional layer, the dimension of the feature map is \(\varphi^{\text{dim}\ _\text{layer}}=3\times 3\times(2^{4+\text{ layer}})\). In the end, the fully connected layer applies the obtained dependency of the input to the output. The model repeats regression steps for each of the outputs being estimated.