# Joint Pedestrian and Body Part Detection via Semantic Relationship Learning
## 3 Our Approach
### Adaptive Joint Non-Maximum Suppression  
Non-Maximum Suppression (NMS) [3] has been an effective post-processing algorithm in proposal based object detection framework. In our joint pedestrian and body part detection task, we notice the traditional NMS could have the issue that it does not treat the individual parts from one pedestrian as a whole, and thus is not able to leverage the detected body parts with high confidence scores to recover the body parts with low detection scores due to occlusion, etc. As a result, there are false positive and false negative detections that can likely be avoided. In order to reduce such false positive and false negative detections, we propose an Adaptive Joint Non-Maximum Suppression (AJ-NMS) to replace the original NMS. Specifically, AJ-NMS considers individual body parts (i.e., head, head-shoulder, upper body, and whole body) as a whole and use adaptive threshold per body part. We provide the details of AJ-NMS in Algorithm 2. AJ-NMS uses the following joint intersection over union (\(IOU_{joint}\)):  
\[IOU_{joint}=\lambda_{1}IOU_{pedestrian}+\lambda_{2}IOU_{head}+\lambda_{3} IOU_{head-shoulder}+\lambda_{4}IOU_{upper-body}, \tag{1}\]  
where: \(IOU_{joint}\) is the joint IoU considering both the holistic pedestrian and the associated body parts. \(IOU_{pedestrian}\), \(IOU_{head}\), \(IOU_{head-shoulder}\) and \(IOU_{upper-body}\) are the traditional IOU defined on pedestrian, head, head-shoulder, and upper-body, respectively. \(\lambda_{1}-\lambda_{4}\) are their weights. We empirically determine the weights per part on the training set by greedy search algorithms. Finally, we choose to use \(\lambda_{1}=0.6\), \(\lambda_{2}=0.2\), \(\lambda_{3}=0.1\) and \(\lambda_{4}=0.1\), respectively.  
A brief analysis why the proposed AJ-NMS should work for the joint pedestrian and body part detection task is that as shown in Figure 1, although the two pedestrians' bodies are seriously occluded, their joint IOU considering both the pedestrian and the body parts according to our AJ-NMS remains lower than the threshold to be suppressed because the overlap between the two pedestrians' heads can be very small. As a result, both pedestrian detections can be retained by the proposed AJ-NMS, avoiding the false negative detections and thus improving the recall rate. We also provide some statistical analysis based on the testing set of the CUHK-SYSU Person Search Dataset. Specifically, there are 14,100 ground-truth pedestrian bounding boxes in the whole test set. Among these bounding boxes, 709 pedestrian bounding boxes, for which the IOUs are greater than 0.4 and will be suppressed by the traditional NMS algorithm (leading to missing detections). By contrast, there are only 454 bounding boxes of pedestrian for which the IOUs are above the threshold and to be suppressed by our AJ-NMS. It is worth noting that our AJ-NMS method does not hurt the detection performance while assures higher recall in detecting overlapped pedestrians. Besides improving the pedestrian detection recall under occlusion, the proposed AJ-NMS can be very helpful in improving the precision of small body part detection, such as the head, leveraging the high confident detection for the larger body parts, such as head-shoulder, upper body, etc.  
In the proposed approach, since the head, head-shoulder, upper body, and pedestrian bounding boxes are jointly regressed based on one proposal by RPN, we can know which body parts belong to the same pedestrian in the detection results. We maintain this part-pedestrian association information in our detection framework, so that AJ-NMS can work correctly for a pedestrian and its parts. The pedestrian-part association can also be very useful for the succeeding pedestrian analysis applications, such as holistic and local based person re-identification.