# Joint Pedestrian and Body Part Detection via Semantic Relationship Learning
## 1 Introduction  
Pedestrian and body part (e.g., upper body, head-shoulder and head) detection has wide potential applications, in person re-identification [1], intelligent surveillance, action recognition from body parts [2], etc. Most of the existing methods focus on detecting the whole pedestrians. The early studies on pedestrian detection used hand-crafted features such as Histogram of Oriented Gradients (HOG) [3], Deformable Part Model (DPM) [4] and Integral Channel Features (ICF) [5; 6]. HOG has much better performances than other hand-crafted features at the time. DPM uses mixtures of multi-scale deformable part models to solve the deformation problem. ICF method achieves competitive results to previous features result but with low computational complexity due to the oriented gradient and colour feature (HOG+LUV) selection by boosted decision forests. However, the hand-crafted features are intuitively designed based on pedestrian shape characteristics, which limits their ability to distinguish between pedestrian and complicated backgrounds. In recent years, deep learning methods, e.g., convolutional neural networks (CNNs) are widely used for pedestrian detection, such as Faster R-CNN [7], PVANet [8], YOLO [9], SSD [10], etc. The deep learning based pedestrian detection methods achieve much better results than the traditionalhand-crafted feature based methods. However, the performance of the existing deep learning based pedestrian detection methods can drop significantly under unconstrained scenarios. As shown in Figure 1, the main challenge is that a number of pedestrians in one image can be very small or occluded in the surveillance scenarios, which leads to insufficient feature map. Some efforts have been made to handle occlusions or small objects during pedestrian detection. Repulsion loss was proposed in [11] to improve pedestrian detection performance by utilizing the repulsion of the surrounding pedestrians in the crowd. Ref. [12] employs an attention mechanism across channels to handle various occlusions. Ref. [13] computes an occlusion-aware detection score based on the part detection confidence scores. In order to handle small objects, Ref. [10] utilizes feature pyramids to enhance the representation of various objects across a large range of scales. Further, Feature Pyramid Networks [14] builds high-level semantic feature maps from all scales to solve the problem of detecting small objects that cannot be well captured by the large-scale feature map. Ref. [15] merges detection results from a set of divided views to improve the detection performance for small objects. In addition, Refs. [16; 17] utilize the relations between objects, and [18; 19; 20] utilize a cascaded network to improve detection performance. Despite the efforts in improving pedestrian detection robustness against occlusion and small scale, joint pedestrian and body part detection is seldom studied, which can be more challenging than detecting pedestrian alone. For example, some pedestrians' heads can be completely obscured, while the other body parts are still visible as shown in Figure 1. Ref. [21] proposes a HeadNet to jointly detect pedestrian and the head, head-shoulder, and upper body in a multi-task way [22; 23] by utilizing the body context information. But the performance of joint pedestrian and body part detection in the case of severe overlap/occlusion, remains unsatisfactory. To this end, we propose a Body Part Indexed Feature (BPIF) to encode the semantic relationship between individual body parts (i.e., head, head-shoulder, upper body, and whole body), leading to improved robustness against partial even full occlusions of the body part. We also propose an Adaptive Joint Non-Maximum Suppression (AJ-NMS) as a replacement of the original NMS algorithms widely used in object detection, leading to higher recall for detection overlapped pedestrians. In summary, the contributions of this work are as follows:  
1.