# Robust Human Pose Estimation Method for Body-to-Body Occlusion Using RGB-D Fusion Neural Network
## 4 Experimental Results
### Performance Evaluation of Human Pose Estimation  
To determine the optimal network configuration, self-comparison experiments are conducted by varying the \(R\) parameter, which specifies how many early stages fuse features from the depth image. The experiments use a test set of 8921 randomly selected samples from the NTU RGB+D 120 dataset. The experimental results are presented in Table 5. The self-comparison experiments indicate that the RGB-D network with \(R=3\) achieves the highest performance across all evaluation metrics, including AP at OKS thresholds of 50 and 75, as well as AP and AR across different person sizes. As \(R\) increases from 1 to 3, the accuracy continuously improves. This indicates that, in the initial stages for extracting local features, the additional use of depth information enables more effective separation of overlapping joints and facilitates the extraction of more precise features. However, when \(R\) is 4 or above, a marked decrease in accuracy is observed. This suggests that, during the later stages for global feature extraction, fusing depth features can adversely affect the representational capabilities of features pertaining to both individual joints and their connectivity. In summary, the 3D topological cues derived from depth images primarily facilitate the representation of local structural features. In contrast, the 2D appearance information contained in color images is generally more influential in modeling global contextual characteristics. We compared the modality fusion in our proposed method with the commonly used fusion strategies of early fusion and late fusion, as shown in Table 6. In early fusion and late fusion, the inputs from RGB and depth images are merged either before or after the initial feature extraction stage, respectively. Early fusion showed the lowest AP and AR scores, likely due to the fundamental differences in the data domains of depth and color images. While depth images primarily provide structural information such as distance and shape, color images offer visual details such as color and texture. When these heterogeneous features are simply combined in the early stages, the network may fail to adequately learn the distinct characteristics of each modality, resulting in degraded performance. On the other hand, late fusion yielded slightly better performance compared to using only RGB images but still demonstrated lower overall accuracy than the proposed method. This suggests that reinforcing feature representations solely through depth images is insufficient for significant improvements. In contrast, our method substantially enhances the feature representations of the joints by progressively fusing features from both modalities at each stage. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \(R\) & \(\mathbf{AP}\) & \(\mathbf{p^{50}}\) & \(\mathbf{p^{75}}\) & \(\mathbf{AP^{M}}\) & \(\mathbf{AP^{L}}\) & \(\mathbf{AR}\) \\ \hline
1 & 0.49 & 0.84 & 0.49 & 0.31 & 0.51 & 0.50 \\
2 & 0.50 & 0.85 & 0.49 & 0.32 & 0.51 & 0.51 \\
3 & 0.51 & 0.86 & 0.51 & 0.33 & 0.52 & 0.52 \\
4 & 0.45 & 0.79 & 0.46 & 0.18 & 0.47 & 0.43 \\
5 & 0.24 & 0.46 & 0.23 & 0.09 & 0.26 & 0.25 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Self-comparison results according to the fusion stage adjustment parameter \(R\). Table 7 presents the experimental results obtained from the test set. These experimental results show that the pose estimation network with the fusion stage parameter \(R\) = 3 achieves an AP improvement of 11.7 and an AR improvement of 13.3 compared to the original OpenPose model. Additionally, as the OKS threshold increases, OpenPose exhibits a significant decline in accuracy and recall, whereas the proposed method demonstrates a relatively lower performance degradation rate. Despite enhancing pose estimation accuracy, the proposed method only increases GPU Floating Point Operations Per Second (GFLOPS) by approximately 44% compared to OpenPose, which is capable of processing approximately 200 images per second in real time.