# Joint Pedestrian and Body Part Detection via Semantic Relationship Learning
## 4 Experiments
### Comparisons with the State-of-Art  
In this section, we provide evaluations of the proposed approach and a number of state-of-the-art algorithms such as Faster R-CNN [7], R-FCN-50 [35], R-FCN-101 [35], PVANet [8], and HeadNet [21] under the same experimental settings, except that Faster R-CNN, R-FCN-50, and R-FCN-101 do not use OHEM and the image shorter side is scaled between 416 pixels and 768 pixels due to memory occupation of R-FCN-101. All evaluation results in terms of recall vs. false positives per image (FPPI) and average precision (AP) are presented in Figure 4 and Table 1. We can observe that although all methods can perform well for the holistic pedestrian detection, the performance of all the methods except for the proposed approach and HeadNet [21] drop significantly in detecting the small body parts. This observation indicates that the spatial semantics of the entire body is useful for a joint pedestrian and body part detection task. Compared with HeadNet, the proposed approach achieves 0.8%, 0.3%, 0.6%, and 1.0% higher AP, and 0.9%, 0.3%, 1.0%, and 1.8% higher recall rate at 1 false positive per image (FPPI) in detecting pedestrian, upper body, head-shoulder, and head detection, respectively. The gain comes from the proposed BPIF which not only encodes the semantic relationship of individual body parts but also highlights the per part features, as well as the proposed AJ-NMS algorithm which jointly considers individual parts instead of doing NMS separately for each part.  
Figure 5 gives some detection results by the proposed approach, we can see the proposed joint pedestrian and body part detection approach performs well under the challenging surveillance scenario with partial occlusions, diverse scales, bad illuminations, etc. In addition, the proposed approach can output the body part and pedestrian association information, i.e., which body parts belong to the same pedestrian.  
Our approach is implemented using Caffe ([[https://github.com/BVLC/caffe](https://github.com/BVLC/caffe)]([https://github.com/BVLC/caffe](https://github.com/BVLC/caffe))), with a few layers in the proposal network implemented in python (the python layers run on CPU). On our platform with a GeForce GTX 1080 Ti GPU and Intel Xeon(R) E5-2620 v4 CPU, the state-of-the-art method HeadNet runs in about 10.8 FPS, and our approach without optimization runs in about 4.5 FPS. The increased computational cost mainly comes from the increased feature map size because of the use of BPIF. In the future, we will optimize the BPIF module to reduce its computational cost, i.e., through Depthwise Separable Convolution [36].  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Pedestrian AP** & **Upper-Body AP** & **Head-Shoulder AP** & **Head AP** \\ \hline Faster R-CNN [7] & 92.1 & 82.3 & 56.2 & 20.3 \\ R-FCN-50 [35] & 91.4 & 79.7 & 54.7 & 17.2 \\ R-FCN-101 [35] & 91.8 & 78.6 & 55.3 & 14.3 \\ PVANet [8] & 90.8 & 89.3 & 85.1 & 64.6 \\ HeadNet [21] & 94.1 & 92.8 & 90.8 & 86.0 \\ Proposed & **94.9** & **93.1** & **91.4** & **87.0** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparisons of detection performance (in terms of AP) for the whole pedestrian, upper-body, head-shoulder, and head by different approaches.  
Figure 4: Detection performance (in terms of recall vs. FPPI) by the proposed approach and the state-of-the-art methods on the CUHK-SYSU Person Search Dataset. (**a**) ROC curve for head detection, (**b**) ROC curve for head-shoulder detection, (**c**) ROC curve for upper body detection, and (**d**) ROC curve for the holistic pedestrian detection.