# Robust Human Pose Estimation Method for Body-to-Body Occlusion Using RGB-D Fusion Neural Network
## 4 Experimental Results
### Dataset  
The NTU RGB+D 120 dataset was adopted to evaluate the performance of the proposed method. Although the NTU RGB+D 120 dataset was primarily developed for benchmarking action recognition tasks, it contains a sufficient number of scenes with body-to-body occlusions with multiple individuals and provides body skeleton data. Therefore, the NTU RGB+D 120 dataset is also suitable for evaluating pose estimation based on RGB-D images. The NTU RGB+D 120 dataset contains RGB-D images, that is, pairs of RGB and depth images, along with corresponding skeleton annotations. This dataset contains approximately 110,000 RGB-D images, along with joint position annotations and 120 action class labels. The RGB-D images were captured using Microsoft Kinect v2. The resolutions of RGB and depth images are 1920 \(\times\) 1080 and 512 \(\times\) 424, respectively. The skeleton annotations include the index corresponding to each type of joint, as defined in Table 2, and the coordinates of each joint in both the RGB and depth images. If a particular joint is not observed due to occlusion or other reasons, the corresponding joint coordinates are filled with zeros. However, these joints that are not visible and, thus, have coordinates filled with zeros are excluded from the evaluation. Although the dataset does not directly provide information such as camera parameters that can be used to align the two images, it is possible to compute the alignment matrix using the pairs of joint coordinates provided in both images. We computed a transformation matrix for each scene based on this alignment method to align the RGB image with the depth image. The weights of the proposed network are initialized using the pretrained weights of the OpenPose model [58]. The OpenPose model was trained using the joint index numbering and types defined by the MS COCO format, as shown in Table 3, which differ from the index numbering and joint types defined in NTU RGB+D 120. To address this issue, we evaluated only the joints that are commonly defined in both formats, which correspond to joints 1 to 14 in the MS COCO format.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Index** & **Joint Type** & **Index** & **Joint Type** \\ \hline
1 & Base of the spine & 14 & Left knee \\
2 & Middle of the spine & 15 & Left ankle \\
3 & Neck & 16 & Left foot \\
4 & Head & 17 & Right hip \\
5 & Left shoulder & 18 & Right knee \\
6 & Left elbow & 19 & Right ankle \\
7 & Left wrist & 20 & Right foot \\
8 & Left hand & 21 & Spine \\
9 & Right shoulder & 22 & Tip of the left hand \\
10 & Right elbow & 23 & Left thumb \\
11 & Right wrist & 24 & Tip of the right hand \\
12 & Right hand & 25 & Right thumb \\
13 & Left hip & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Indices of joint types in NTU RGB+D 120 dataset.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Index** & **Joint Type** & **Index** & **Joint Type** \\ \hline
1 & Nose & 10 & Right knee \\
2 & Neck & 11 & Right ankle \\
3 & Right shoulder & 12 & Left hip \\
4 & Right elbow & 13 & Left knee \\
5 & Right wrist & 14 & Left ankle \\
6 & Left shoulder & 15 & Right eye \\
7 & Left elbow & 16 & Left eye \\
8 & Left wrist & 17 & Right ear \\
9 & Right hip & 18 & Left ear \\ \hline \hline \end{tabular}
\end{table}
Table 3: Indices of joint types in MS COCO format.