# Robust Human Pose Estimation Method for Body-to-Body Occlusion Using RGB-D Fusion Neural Network
## 3 Human Pose Estimation Method by Progressive Feature Fusion
### Network Architecture  
The proposed method effectively considers the topological relationships between detected candidate joints using three-dimensional information. This allows for accurate joint detection even in occluded scenes. In this work, we adopt OpenPose [30] as our HPE baseline model due to its widespread use and proven effectiveness. Figure 2 illustrates the structure of the proposed network. The network consists of three parallel branches, each containing independent inference layers. The first branch extracts PAFs, while the second and third branches generate confidence maps for body joints using RGB and depth images, respectively. The network employs a multi-stage  
Figure 1: Flowchart of the proposed method. cascaded architecture, where the outputs of each branch are fused and used as input for the subsequent stage. This iterative process progressively expands the receptive field and refines feature representations, ultimately improving the final output. The proposed network is based on the OpenPose model and adopts the first ten layers of VGG-19 [40] as the backbone to extract feature maps \(\mathbf{F}_{rgb}\) and \(\mathbf{F}_{d}\) from RGB and depth images. In the first stage, three independent inference layers, \(\phi^{1}\), \(\rho^{1}\), and \(\gamma^{1}\), predict pose-related information. The inference layers \(\phi^{1}\) and \(\rho^{1}\) take \(\mathbf{F}_{rgb}\) as input, while \(\gamma^{1}\) processes \(\mathbf{F}_{d}\). The inference layer \(\phi^{1}\) extracts a two-dimensional vector field \(\mathbf{L}^{1}\), encoding the directional information of joint connections at limb locations. The inference layer \(\rho^{1}\) generates a confidence map \(\mathbf{S}^{1}\), predicting body joint locations from the color feature maps. The inference layer \(\gamma^{1}\) produces a confidence map \(\mathbf{D}^{1}\), estimating body joint positions using depth feature maps. All three inference layers, \(\phi^{1}\), \(\rho^{1}\), and \(\gamma^{1}\), have an identical structure consisting of three convolutional layers with a \(3\times 3\) kernel and two convolutional layers with a \(1\times 1\) kernel. The detailed structure of these layers is shown in Figure 3. After extracting \(\mathbf{L}^{1}\), \(\mathbf{S}^{1}\), and \(\mathbf{D}^{1}\) from each branch in Stage 1, the confidence maps \(\mathbf{S}^{1}\) and \(\mathbf{D}^{1}\) are fused. A convolutional layer with a \(1\times 1\) kernel is then applied to reduce the number of channels by half, resulting in the fused confidence map \(\mathbf{M}^{1}\), which integrates both color and depth information. Similarly, the feature maps \(\mathbf{F}_{rgb}\) and \(\mathbf{F}_{d}\) are combined, and a \(1\times 1\) convolutional layer is used to reduce the number of channels by half, producing \(\mathbf{F}_{rgb}\), which encapsulates both color and depth features. The fused outputs \(\mathbf{M}^{1}\), \(\mathbf{L}^{1}\), and \(\mathbf{F}_{rgb}\) are then combined and used as input for each branch in Stage 2. For Stage \(n\) (\(n\geq 2\)), the inference layers in each branch consist of five sequential convolutional layers with a \(7\times 7\) kernel, followed by two convolutional layers with a \(1\times 1\) kernel. Figure 4 illustrates the structure of the inference layers \(\phi^{n}\), \(\rho^{n}\), and \(\gamma^{n}\) for each branch in Stage \(n\). Figure 3: Structure of inference layer in Stage 1. Figure 2: Structure of the proposed network. The proposed RGB-D network fuses color and depth information only up to Stage \(R\). In these stages, the fused confidence map \(\mathbf{M}^{\prime}\) is obtained by combining \(\mathbf{S}^{\prime}\) and \(\mathbf{D}^{\prime}\), while \(\mathbf{L}^{\prime}\) and \(\mathbf{F}_{rgbd}\) are also incorporated as inputs for the next stage. After Stage \(R\), the network no longer includes Branch 3, which processes depth information. Instead, the subsequent stages use only \(\mathbf{L}^{\prime}\), \(\mathbf{S}^{\prime}\), and \(\mathbf{F}_{rgb}\) as inputs.