# Robust Human Pose Estimation Method for Body-to-Body Occlusion Using RGB-D Fusion Neural Network
## 1 Introduction  
Human pose estimation (HPE) detects body joints from images and estimates the human skeleton. HPE is widely used in HAR (Human Action Recognition), which recognizes body posture or movements [1, 2, 3], because it is not affected by lighting changes or background complexity [4]. Vision-based HAR utilizing HPE has diverse applications in the field of computer vision, including surveillance video analysis and human-computer interaction (HCI) [5, 6, 7, 8, 9]. Recently, a fine-grained pose estimation method using multi-modal information has also been actively studied for HCI [10, 11]. HPE methods can be categorized into top-down and bottom-up approaches based on the sequence of body detection and joint localization [12, 13, 14, 15]. The top-down approach first detects human regions in an image and then applies a joint detector to each region. Thismethod is less sensitive to variations in human scale but suffers from a linear increase in computational cost as the number of people in an image increases. In contrast, the bottom-up approach directly detects body joints in an image and estimates their associations based on spatial relationships to reconstruct human poses. This approach is advantageous for real-time processing since the joint detector operates only once [16]. Although recent HPE models [17; 18; 19; 20; 21] have achieved outstanding average accuracy and inference speed, an occlusion problem, which refers to a drop in accuracy when some body parts are obstructed by other parts or objects, still remains to be solved. The occlusion problem can be classified into occlusion by objects, body-to-body occlusion, and self-occlusion (e.g., when a person crosses his or her legs). Improvements in HPE for images with occlusion by objects have been studied, whereas other types of occlusion problems have not been adequately addressed [22]. These occlusion problems are mainly caused by the lack of topological information in RGB images [23; 24]. In this study, we focus on the improvement in HPE for images with body-to-body occlusion. There are efforts to estimate multiple people's poses from images captured from multiple views to reduce the influence of occlusion [25]. In this case, it can be mostly free from occlusion, and even 3D pose estimation can be considered. However, this approach is limited in that it can only be used in controlled environments where multiple cameras are precisely installed. RGB-D images that contain both color and depth images can capture topological relationships in three-dimensional space. A depth image stores distances from a camera device along the \(z\)-axis, which is the camera optical axis, as its pixels. By additionally utilizing the pixel values of the depth image, occluded body parts can be more effectively separated. Previous studies [26; 27; 28] have been conducted to estimate object poses by fusing features from color and depth images. Nevertheless, these studies for HPE have typically employed depth images only for auxiliary tasks, such as body region segmentation or post-processing corrections of detected skeletal data, rather than integrating depth information directly into the network itself. In particular, few studies have focused on leveraging depth images to address body-to-body occlusion in multi-person images. We aim to improve HPE for occluded bodies by fusing features from both color and depth images. In this study, we propose an HPE method robust to body-to-body occlusion problems using a novel network architecture that integrates color and depth information. We modify the traditional OpenPose that is a bottom-up HPE model to take a depth image as an additional input, thereby providing explicit 3D spatial cues. Each input modality is processed by a dedicated feature extractor. The proposed network consists of a multi-stage cascade structure and acts as a joint detection method that effectively considers both color and depth information by iteratively fusing features extracted from two inputs.