# Robust Human Pose Estimation Method for Body-to-Body Occlusion Using RGB-D Fusion Neural Network
## 3 Human Pose Estimation Method by Progressive Feature Fusion
### Ground Truth for Heatmap Representation  
In the proposed network, the GT for network training consists of confidence maps \(\mathbf{Q}^{*}\) that represent joint locations as heatmaps and a set of 2D vectors \(\mathbf{L}^{*}\) that encode the directional information between connected joints. The generation of GT from the annotated body joint coordinates follows the procedures described in OpenPose. When training body joint localization, the confidence maps have a width \(w\) and height \(h\), and the ground truth \(\mathbf{Q}^{*}\in\mathbb{R}^{j\times w\times h}\) is defined as a set of heatmaps \(\mathbf{Q}^{*}_{j}\in\mathbb{R}^{w\times h}\) for each joint \(j\). To generate \(\mathbf{Q}^{*}_{j}\), the function \(\mathbf{Q}^{*}_{j,k}\in\mathbb{R}^{w\times h}\) must first be computed. The function \(\mathbf{Q}^{*}_{j,k}\) represents a heatmap modeled using a Gaussian distribution with variance \(\sigma^{2}\), based on the distance between a pixel \(\mathbf{p}\) and the actual joint coordinate \(\mathbf{x}_{j,k}\) of person \(k\) in the dataset. The function \(\mathbf{Q}^{*}_{j,k}\) is computed as follows:  
\[\mathbf{Q}^{*}_{j,k}(\mathbf{p})=\exp\left(-\frac{\left\|\mathbf{p}-\mathbf{x }_{j,k}\right\|_{2}^{2}}{\sigma^{2}}\right), \tag{5}\]  
where \(\sigma\) controls the spread of the peak in the confidence map represented as a heatmap. In the proposed method, \(\sigma\) is set to 3. The GT of confidence map \(\mathbf{Q}^{*}_{j}\) is generated by applying a maximum operation over all \(\mathbf{Q}^{*}_{j,k}\) at \(\mathbf{p}\) as follows:  
\[\mathbf{Q}^{*}_{j}(\mathbf{p})=\underset{k}{max}\mathbf{Q}^{*}_{j,k}(\mathbf{p }). \tag{6}\]  
Finally, the confidence maps corresponding to the joints are assigned the GT values. The generation of the GT of \(\mathbf{L}^{*}\) requires one to obtain \(\mathbf{L}^{*}_{c,k}\in\mathbb{R}^{w\times h\times 2}\), which represents the connection direction of limb type \(c\) at pixels where the limb of person index \(k\) is present. The vector field \(\mathbf{L}^{*}_{c,k}\) is computed as follows:\[\mathbf{L}_{c,k}^{*}(\mathbf{p})=\left\{ \begin{array}{ll}\mathbf{v}&\text{if $ \mathbf{p}$ on limb $c$, $k$}\\ 0&\text{otherwise}\end{array} \right., \tag{7}\]  
where \(\mathbf{v}\) represents the direction of the limb. If pixel \(\mathbf{p}\) corresponds to the location of the \(c\)-th limb of the \(k\)-th person, \(\mathbf{v}\) is assigned as a unit vector indicating the limb's direction. Otherwise, for non-limb pixels, \(\mathbf{v}\) is set to zero. The vector \(\mathbf{v}\) is computed as follows:  
\[\mathbf{v}=\left(\mathbf{x}_{j_{2},k}-\mathbf{x}_{j_{1},k}\right)/\left\| \mathbf{x}_{j_{2},k}-\mathbf{x}_{j_{1},k}\right\|_{1}, \tag{8}\]  
where \(\mathbf{x}_{j_{1},k},\mathbf{x}_{j_{2},k}\in\mathbb{R}^{2}\) represent the positions of the two joints forming the \(c\)-th limb of the \(k\)-th individual. A unit vector with a magnitude of 1 and a direction from \(\mathbf{x}_{j_{1},k}\) to \(\mathbf{x}_{j_{2},k}\) can be obtained through computation. To assign \(\mathbf{v}\) at limb locations, it is necessary to define the pixel region where the limb exists. Since limbs have thickness, the reference region should include both the line segment connecting the two joints and the area perpendicular to this segment. The threshold for defining the region considering limb thickness is determined based on \(\mathbf{x}_{j_{1},k}\) and \(\mathbf{x}_{j_{2},k}\) and is computed as follows:  
\[0\leq\mathbf{v}\cdot\left(\mathbf{p}-\mathbf{x}_{j_{1},k}\right)\leq l_{c,k} \text{ and }\left|\mathbf{v}_{\perp}\cdot\left(\mathbf{p}-\mathbf{x}_{j_{2},k} \right)\right|\leq\sigma_{l}, \tag{9}\]  
where \(\mathbf{v}_{\perp}\) is a unit vector perpendicular to the direction of \(\mathbf{v}\), \(\sigma_{l}\) represents a limb width that is a distance in pixels, and \(l_{c,k}\) denotes the limb length of the \(c\)-th limb for \(k\)-th person.