# A Review of Intelligent Driving Pedestrian Detection Based on Deep Learning

Authors: Di Tian; Yi Han; Biyao Wang; Tian Guan; Wei Wei
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Pedestrian detection is a specific application of object detection. Compared with general object detection, it shows similarities and unique characteristics. In addition, it has important application value in the fields of intelligent driving and security monitoring. In recent years, with the rapid development of deep learning, pedestrian detection technology has also made great progress. However, there still exists a huge gap between it and human perception. Meanwhile, there are still a lot of problems, and there remains a lot of room for research. Regarding the application of pedestrian detection in intelligent driving technology, it is of necessity to ensure its real-time performance. Additionally, it is necessary to lighten the model while ensuring detection accuracy. This paper first briefly describes the development process of pedestrian detection and then concentrates on summarizing the research results of pedestrian detection technology in the deep learning stage. Subsequently, by summarizing the pedestrian detection dataset and evaluation criteria, the core issues of the current development of pedestrian detection are analyzed. Finally, the next possible development direction of pedestrian detection technology is explained at the end of the paper.


---

## Body

## 1. Introduction

Object detection is a basic problem of machine vision and deep learning, and it lays the basis for the in-depth development of numerous research problems, including instance segmentation [1–3], object tracking and optimization [4–6], trajectory prediction [7], and image reconstruction [8–10]. Pedestrian detection is a specific application of the object detection problem, and it has become one of the research hotspots in recent years. It has important application value in the fields of intelligent driving and security monitoring. Particularly in the field of intelligent driving, due to the particularity of people and the highest safety requirements, it is more important than other types of object detection. In intelligent driving, the camera, lidar [11–13], and wireless sensor network [14–18] jointly perceive the environment and further employ vehicle-mounted computers and cloud computing [19–23] to make decisions and control. Figure 1 presents the trend of the number of publications in association with pedestrian detection in recent years. Compared with other types of object detection, pedestrian detection puts forward stricter requirements on accuracy and real-time performance, which is of extraordinary significance in the field of intelligent driving. In recent years, large quantities of reviews of general object detection have been published [24–28], but there are few reviews of pedestrian detection, lacking the analysis of its latest developments and discussion of current difficulties. By performing a rough analysis of general object detection, this paper will discuss in-depth pedestrian detection.Figure 1
Number of publications related to pedestrian detection in recent years.The development of object detection tasks has mainly experienced two major stages, respectively, the traditional object detection period and the detection period based on deep learning. As early as 2001, P. Viola and M. Jones proposed the famous VJ detector [29]. It combines a variety of important technologies such as “integral image,” which significantly improves the detection efficiency and detection capabilities and realizes the real-time detection of fixed object for the first time, strongly promoting the development of the object detection field. In particular, in 2005, Dalal and Triggs proposed the Histogram of Oriented Gradients (HOG) feature descriptors [30], which designed the HOG descriptors to be calculated on a dense uniformly spaced cell grid and adopted overlapping local contrast to normalize in order to improve the accuracy. Although HOG can be used to detect various object classes, its main research goal is to solve the problem of pedestrian detection. The proposed method has achieved a very high accuracy rate, which strongly demonstrates the effectiveness of this algorithm. Subsequently, in order to promote the development of the field of pedestrian detection, the INRIA pedestrian dataset, which is still widely used, was published. Later in 2008, Felzenszwalb proposed the DPM detection algorithm [31], which can divide pedestrian into different parts for training and learning as well as treating them as a collection of different parts detection during classification. Under this kind of thinking, the algorithm and its improved algorithm have continuously obtained the best detection results for several years, reaching the relative peak of traditional detection algorithms. Additionally, there are scholars who study general computer vision methods, which can improve various computer vision problems [32–37].The implementation process of the traditional object detection method is similar to the VJ detector. It mainly extracts object features through artificial design (such as HOG, Haar, and SIFT) and new feature extraction methods [38, 39] and further uses SVM, DT [40], and other classifiers for recognition and detection. Before the detection, the image is often preprocessed to enhance the image quality [41–43]. In the detection process, sliding window processing is usually performed on the image to predict the object. At that time, the best detection performance is achieved. However, because the sliding window method traverses all possible positions and size ratios, it places high requirements on the computing power of the computer. In addition, the hand-designed feature expression ability is weak, contributing to a poor overall detection effect. In 2014, Girshick et al. proposed the RCNN algorithm [44] for feature extraction using CNN. This algorithm vigorously stimulated the development of object detection tasks and advanced it to the development stage of deep learning. In general, deep learning can use the gradient descent method to automatically optimize model parameters [45]. Various object detection tasks have achieved leap-forward progress. Later, some optimization methods in association with neural networks appeared [46–48]. At present, neural networks have a wide range of applications [49–53]. The development process of pedestrian detection is displayed in Figure 2.Figure 2
The development process of pedestrian detection.The rest of the work is arranged as follows. In the second part, the current mainstream pedestrian detection algorithms are summarized. In the third part, the commonly used datasets and evaluation methods in the pedestrian detection field are presented. In the fourth part, the occlusion problem and the multiscale problem that affect the pedestrian detection effect are analyzed in detail. The full text is summarized and prospected in the fifth part.


## 2. Pedestrian Detection Method Based on Deep Learning

Since Girshick et al. proposed RCNN in 2014, the task of pedestrian detection has officially entered the deep learning stage. In general, detection methods based on deep learning mainly consist of two categories. One is a two-stage processing method. Firstly, regional suggestion boxes for possible object are generated, and then further predictions are made on these suggestion boxes. The other is a one-stage processing method, which directly returns the object area on the feature map and gives the final prediction result. The following part summarizes the specific applications of these two detection frameworks in pedestrian detection.


### 2.1. Two-Stage Detection Framework

The two-stage detection framework is mainly divided into two stages: region suggestion and object detection. First, a series of region suggestion boxes are proposed on the image to be inspected. Then, object detection is further conducted. The RCNN detection framework proposed by R. Girshick in 2014 first uses selective search [54] to generate a region suggestion box on the image, then uses CNN for feature extraction, further trains the SVM classifier and bounding-box regression, and finally predicts the result. Although the use of CNN for feature extraction greatly improves the detection effect, it also encounters many problems, such as cumbersome training process and long detection time. Subsequently, improved Fast RCNN [55] and Faster RCNN [56] algorithms are proposed to address the above problems. Faster RCNN completes the end-to-end detection process. First, the RPN algorithm is proposed to replace the selective search for regional recommendation, which greatly reduces the time consumed by regional recommendations. In addition, shared features help avoid repeated feature calculations, the detection accuracy on the VOC07 dataset [57, 58] reaches 73.2%, and the detection accuracy on the COCO dataset [59] reaches 42.7%. The framework diagram of the Faster RCNN series of algorithms is shown in Figure 3.Figure 3
Faster RCNN series algorithm framework diagram [44, 55, 56]. (a) RCNN. (b) Fast RCNN. (c) Faster RCNN.
(a)(b)(c)In 2015, Cai et al. deduced that the Comp ACT algorithm [60] not only optimizes classification risk but also better combines feature extraction and classifier function, which plays an important role in promoting pedestrian classification at different scales. In 2016, Dai et al. made a series of improvements based on Faster RCNN and proposed RFCN [61]. RFCN integrates location information into the pooling layer, enhances location sensitivity, and improves the processing results of pedestrian detection problems that are more sensitive to location information. Compared with Faster RCNN, the introduction of FCN achieves more network parameters and feature sharing, reduces the amount of repetition in the network, and improves the running speed. In 2017, the Mask RCNN proposed by Kaiming He et al. adds a convolutional layer after the pooling layer to perform mask prediction tasks. This structure can complete tasks such as pedestrian detection and pedestrian segmentation and separate pedestrians from the background. At the same time, the result can be further used for human body gesture recognition.In 2017, Lin et al. proposed FPN [62] based on Faster RCNN. Before that, most of the detectors were detected at the top of the network. Although it has good semantic information for category detection, it is not conducive to pedestrian positioning due to the small feature map. FPN proposes a top-down prediction structure and builds high-level semantic information on the entire convolution structure, making pedestrian detection greatly improved.In 2018, Li et al. proposed SAF RCNN based on the perception theory [63], which effectively improved the performance of pedestrian detection of different scales.Among the two-stage detection methods based on deep learning mentioned above, the RCNN series methods (RCNN [44], Fast RCNN [55], and Faster RCNN [56]) are the earliest ones proposed in recent years. The RCNN series method is a general object detection method, which is not specially optimized for a typical category and can be used in various object detection tasks. The main constraints on the performance of RCNN and Fast RCNN are repeated convolution calculations and region proposal networks, which have been improved in Faster RCNN and achieved the best results at the time. The Comp ACT algorithm [60] introduced above is mainly used in the field of pedestrian detection. This algorithm can improve the processing capacity of pedestrian detection and can be extended to other object detection problems to a certain extent. The RFCN [61] algorithm is mainly proposed for general object detection, and it can also achieve good results in specific pedestrian detection areas. Mask RCNN [3] is an improvement based on Faster RCNN. It is a solution proposed for general object detection, and it also has a good effect in the field of pedestrian detection. The FPN [62] algorithm constructs a feature pyramid network, which greatly improves the general object detection and pedestrian detection problems. The SAF RCNN [64] algorithm is mainly used in the field of pedestrian detection in natural scenes. It can also improve the general object detection ability, but because the object scale change is more common in the field of pedestrian detection, the improvement in general object detection is limited. Table 1 summarizes the calculation speeds of the two-stage detection methods mentioned above.Table 1
Calculation speed of some two-stage algorithms.
MethodModelRate (fps)RCNN [44]AlexNet13 (s)Fast RCNN [55]VGG2 (s)Faster RCNN [56]VGG5Comp ACT [60]VGG2.5RFCN [61]ResNet5.9Mask RCNN [3]ResNet-FPN5FPN [62]ResNet5SAF RCNN [64]VGG1.7There are two parts in the two-stage pedestrian detection framework: region suggestion and classification. Researchers can improve the detection effect by proposing different preselection box generation algorithms and feature extraction algorithms or improve the detection results by enhancing the prediction part. Although the overall framework is more cumbersome than the one-stage framework, it has better robustness and accuracy overall.


### 2.2. One-Stage Detection Framework

Compared with the two-stage detection framework, the one-stage detection framework removes the preselection box generation algorithm and directly predicts the object center and object bounding box by setting a series of anchors on the feature map. In 2015, Redmon et al. proposed the first single-stage detector YOLO [65] in the deep learning era. The idea of this detector is shown in Figure 4. It applies a single neural network to the entire image and divides the image into multiple regions. This mode greatly improves the detection speed while predicting the bounding box and probability of each region simultaneously. In the task of pedestrian detection, especially in the pedestrian detection of intelligent driving technology, the detection speed is particularly important [66]. Only high-speed detection can avoid a series of hazards. The one-stage detection framework provides the possibility for this.Figure 4
Algorithm idea of YOLO detector [65].Compared with the two-stage detector, the positioning accuracy of YOLO has decreased, and because it only predicts a limited number of objects at a prediction anchor, the detection effect for small objects and group objects is poor. In response to the above problems, J. Redmon proposed YOLOv2 [67] and YOLOv3 [68]. They were optimized for the above problems, which not only greatly improved the detection accuracy of the one-stage detector but also achieved a relative balance between speed and accuracy. Particularly for YOLOv3, three prediction channels are used to improve the effect of multiscale prediction in pedestrian detection. The structural frame diagram of YOLOv3 is displayed in Figure 5.Figure 5
Framework diagram of YOLOv3.In 2016, Liu et al. further proposed an SSD one-stage detection framework [69]. Unlike YOLO, the SSD algorithm outputs feature layers of different sizes through multilayer mapping in the convolutional layer to detect multiscale objects. In particular, the detection effect of small objects is improved.In 2017, Lin et al. proposed the RetinaNet detector [70]. In response to the poor detection effect of the one-stage detector, a new loss function is introduced in it, so that the detector pays more attention to the difficulty in classifying samples during the training process and solves the problem of unbalanced samples in the work of the one-stage detector. Overall, the single-stage detector can improve its detection accuracy while maintaining a high detection speed. In 2018, Liu et al. put forward an efficient one-stage pedestrian detection architecture ALFnet [71], which mainly uses the continuously increasing IOU threshold to train multiple positioning modules. It can improve the detection accuracy of pedestrian detection. It can achieve the same detection speed as SSD and the same detection accuracy as Faster RCNN; at that time, the most advanced performance was achieved on the CityPersons dataset and the Caltech dataset. In 2019, Zheng et al. proposed DIOU Loss and CIOU Loss [72] to optimize the previous loss function. Compared with the previous object box regression loss, it considers the overlap area, center point distance, and aspect ratio. The bounding box considering the distance loss has faster convergence speed and higher convergence accuracy, which improves the detection accuracy of the object detection framework.In 2020, Alexey Bochkovskiy proposed YOLOv4 [73]. Based on the advantages of multiple detection frameworks, Backbone partially uses the CSPNet structure [74] proposed by Wang et al. in 2020. The schematic diagram of applying CSP to ResNet is shown in Figure 6, which adds a path to each cycle block. In the neck part, the feature fusion is performed by adding the SPP structure [75] and the PAN structure [76]. In addition, the advantages of clustering [77] are used to generate the predicted frame size. The SPP structure can help the network integrate the features of different scales, and the PAN structure integrates the features obtained from different layers. Finally, YOLOv4 obtains 65.7% (AP50) detection accuracy and 65FPS detection speed on the coco dataset, achieving the best balance between the current detection frame speed and accuracy. In addition, some scholars study the application of object detection to slam so as to promote the development of related technologies [78]. So far, pedestrian detection algorithms have mostly focused on the two-stage network framework. However, pedestrian detection in intelligent driving technology has high requirements for real-time performance. With the breakthroughs in accuracy and real-time performance of frameworks such as YOLOv4, pedestrian detection technology in the future will focus more on the single-stage detection framework. Beyond that, these algorithms have also laid a solid foundation for the application of pedestrian detection in intelligent driving.Figure 6
CSPNet structure [74].Among the above-mentioned one-stage detection methods based on deep learning, the YOLO series methods (YOLOv1 [65], YOLOv2 [67], YOLOv3 [68], and YOLOv4 [73]) are the earliest ones proposed in recent years. The algorithms of the YOLO series can be used in various object detection tasks. Due to the reason that only a limited number of objects are predicted in one anchor, it often causes missed detection in the scene of crowded pedestrians, so the performance of the algorithm will be reduced in the crowded scene. However, the high detection speed of such algorithms provides the possibility for the application of pedestrian detection technology in the field of intelligent driving. The SSD [69] algorithm mentioned above is proposed for general object detection, which can improve the problem of multiscale detection in pedestrian detection. The RetinaNet [70] detector introduces a new loss function, which can improve the detection accuracy in the general object detection field. The ALFnet [71] algorithm is mainly used for pedestrian detection. Due to the effective improvement of the task of pedestrian detection, it can be extended to general object detection to a certain extent. The CIOU Loss [72] algorithm researches the boundary regression problem in object detection, which effectively improves the detection effect of various objects. Table 2 summarizes the calculation speeds of the one-stage detection methods mentioned above.Table 2
Calculation speed of some one-stage algorithms.
MethodBackboneRate (fps)YOLOv1 [65]—45YOLOv2 [67]Darknet40YOLOv3 [68]Darknet34.5YOLOv4 [73]Darknet65SSD [69]VGG59RetinaNet [70]ResNet-FPN13.7ALFnet [71]ResNet3.7


### 2.3. Backbone

The pedestrian detection algorithms are different. However, in the deep learning stage, the first is to use the convolutional neural network to process the image to obtain the deep feature map and then perform various subsequent processing. This part obtains the convolutional neural network of the feature map called the “Backbone” of the entire algorithm. Backbone can decisively influence the effect of the network. This section will review this content.


#### 2.3.1. VGGNet

After AlexNet [79] achieves excellent results in the ImageNet competition, the VGGNet [80] proposed by Simonyan in 2014 improves the convolutional neural network, uses a smaller convolution kernel and a deeper network structure, and achieves better results.


#### 2.3.2. INception

In the process of extracting features of the convolutional neural network, increasing the depth and width of the network can improve the performance of the network. Nonetheless, doing so will also lead to a substantial increase in the number of parameters, and it is prone to overfitting. Inception [81], proposed in 2014, solves this problem better. It uses three convolution kernels of different sizes for convolution calculations and then cascades these parts to enter the next layer. Later, the improved v2, v3, and v4 versions [82–84] are proposed.


#### 2.3.3. ResNet

Based on VGGNet and Inception, He et al. proposed ResNet [85] in 2015, solving the problem of gradient disappearance and gradient update difficulty. Since then, ResNet has been generally used as Backbone for various classification, detection, and segmentation tasks. The main idea is to introduce a residual block, let the convolutional network learn the residual mapping, and make the network optimization easier.


#### 2.3.4. DenseNet

In 2017, DenseNet [86] maximized the information exchange between the front and rear layers based on ResNet. By establishing dense connections between all the front layers and all the back layers, it realizes the multiplexing of features in the channel dimension. This structure can achieve better performance than ResNet with fewer parameters and calculations.


#### 2.3.5. FPN

In order to enhance semantics, traditional object detection models usually only perform follow-up operations on the last feature layer, but the final feature map often has less detailed information, making the detection of small objects more difficult. In 2017, the FPN method merged the features of different layers, which better improves the multiscale detection problem. The overall architecture of FPN mainly consists of four parts: bottom-up network, top-down network, horizontal connection, and convolution.


#### 2.3.6. DetNet

DetNet [87] introduces the hole convolution, which increases the receptive field, obtains a larger feature map size, and makes the model have a larger receptive field and higher resolution. At the same time, the detection of large objects and small objects is taken into account. It is especially suitable for inspection tasks. The structure diagram is shown in Figure 7.Figure 7
DetNet structure diagram [87]. (a) Dilated bottleneck. (b) Dilated bottleneck with 1 × 1 conv projection. (c) Original bottleneck.
(a)(b)(c)


## 2.1. Two-Stage Detection Framework

The two-stage detection framework is mainly divided into two stages: region suggestion and object detection. First, a series of region suggestion boxes are proposed on the image to be inspected. Then, object detection is further conducted. The RCNN detection framework proposed by R. Girshick in 2014 first uses selective search [54] to generate a region suggestion box on the image, then uses CNN for feature extraction, further trains the SVM classifier and bounding-box regression, and finally predicts the result. Although the use of CNN for feature extraction greatly improves the detection effect, it also encounters many problems, such as cumbersome training process and long detection time. Subsequently, improved Fast RCNN [55] and Faster RCNN [56] algorithms are proposed to address the above problems. Faster RCNN completes the end-to-end detection process. First, the RPN algorithm is proposed to replace the selective search for regional recommendation, which greatly reduces the time consumed by regional recommendations. In addition, shared features help avoid repeated feature calculations, the detection accuracy on the VOC07 dataset [57, 58] reaches 73.2%, and the detection accuracy on the COCO dataset [59] reaches 42.7%. The framework diagram of the Faster RCNN series of algorithms is shown in Figure 3.Figure 3
Faster RCNN series algorithm framework diagram [44, 55, 56]. (a) RCNN. (b) Fast RCNN. (c) Faster RCNN.
(a)(b)(c)In 2015, Cai et al. deduced that the Comp ACT algorithm [60] not only optimizes classification risk but also better combines feature extraction and classifier function, which plays an important role in promoting pedestrian classification at different scales. In 2016, Dai et al. made a series of improvements based on Faster RCNN and proposed RFCN [61]. RFCN integrates location information into the pooling layer, enhances location sensitivity, and improves the processing results of pedestrian detection problems that are more sensitive to location information. Compared with Faster RCNN, the introduction of FCN achieves more network parameters and feature sharing, reduces the amount of repetition in the network, and improves the running speed. In 2017, the Mask RCNN proposed by Kaiming He et al. adds a convolutional layer after the pooling layer to perform mask prediction tasks. This structure can complete tasks such as pedestrian detection and pedestrian segmentation and separate pedestrians from the background. At the same time, the result can be further used for human body gesture recognition.In 2017, Lin et al. proposed FPN [62] based on Faster RCNN. Before that, most of the detectors were detected at the top of the network. Although it has good semantic information for category detection, it is not conducive to pedestrian positioning due to the small feature map. FPN proposes a top-down prediction structure and builds high-level semantic information on the entire convolution structure, making pedestrian detection greatly improved.In 2018, Li et al. proposed SAF RCNN based on the perception theory [63], which effectively improved the performance of pedestrian detection of different scales.Among the two-stage detection methods based on deep learning mentioned above, the RCNN series methods (RCNN [44], Fast RCNN [55], and Faster RCNN [56]) are the earliest ones proposed in recent years. The RCNN series method is a general object detection method, which is not specially optimized for a typical category and can be used in various object detection tasks. The main constraints on the performance of RCNN and Fast RCNN are repeated convolution calculations and region proposal networks, which have been improved in Faster RCNN and achieved the best results at the time. The Comp ACT algorithm [60] introduced above is mainly used in the field of pedestrian detection. This algorithm can improve the processing capacity of pedestrian detection and can be extended to other object detection problems to a certain extent. The RFCN [61] algorithm is mainly proposed for general object detection, and it can also achieve good results in specific pedestrian detection areas. Mask RCNN [3] is an improvement based on Faster RCNN. It is a solution proposed for general object detection, and it also has a good effect in the field of pedestrian detection. The FPN [62] algorithm constructs a feature pyramid network, which greatly improves the general object detection and pedestrian detection problems. The SAF RCNN [64] algorithm is mainly used in the field of pedestrian detection in natural scenes. It can also improve the general object detection ability, but because the object scale change is more common in the field of pedestrian detection, the improvement in general object detection is limited. Table 1 summarizes the calculation speeds of the two-stage detection methods mentioned above.Table 1
Calculation speed of some two-stage algorithms.
MethodModelRate (fps)RCNN [44]AlexNet13 (s)Fast RCNN [55]VGG2 (s)Faster RCNN [56]VGG5Comp ACT [60]VGG2.5RFCN [61]ResNet5.9Mask RCNN [3]ResNet-FPN5FPN [62]ResNet5SAF RCNN [64]VGG1.7There are two parts in the two-stage pedestrian detection framework: region suggestion and classification. Researchers can improve the detection effect by proposing different preselection box generation algorithms and feature extraction algorithms or improve the detection results by enhancing the prediction part. Although the overall framework is more cumbersome than the one-stage framework, it has better robustness and accuracy overall.


## 2.2. One-Stage Detection Framework

Compared with the two-stage detection framework, the one-stage detection framework removes the preselection box generation algorithm and directly predicts the object center and object bounding box by setting a series of anchors on the feature map. In 2015, Redmon et al. proposed the first single-stage detector YOLO [65] in the deep learning era. The idea of this detector is shown in Figure 4. It applies a single neural network to the entire image and divides the image into multiple regions. This mode greatly improves the detection speed while predicting the bounding box and probability of each region simultaneously. In the task of pedestrian detection, especially in the pedestrian detection of intelligent driving technology, the detection speed is particularly important [66]. Only high-speed detection can avoid a series of hazards. The one-stage detection framework provides the possibility for this.Figure 4
Algorithm idea of YOLO detector [65].Compared with the two-stage detector, the positioning accuracy of YOLO has decreased, and because it only predicts a limited number of objects at a prediction anchor, the detection effect for small objects and group objects is poor. In response to the above problems, J. Redmon proposed YOLOv2 [67] and YOLOv3 [68]. They were optimized for the above problems, which not only greatly improved the detection accuracy of the one-stage detector but also achieved a relative balance between speed and accuracy. Particularly for YOLOv3, three prediction channels are used to improve the effect of multiscale prediction in pedestrian detection. The structural frame diagram of YOLOv3 is displayed in Figure 5.Figure 5
Framework diagram of YOLOv3.In 2016, Liu et al. further proposed an SSD one-stage detection framework [69]. Unlike YOLO, the SSD algorithm outputs feature layers of different sizes through multilayer mapping in the convolutional layer to detect multiscale objects. In particular, the detection effect of small objects is improved.In 2017, Lin et al. proposed the RetinaNet detector [70]. In response to the poor detection effect of the one-stage detector, a new loss function is introduced in it, so that the detector pays more attention to the difficulty in classifying samples during the training process and solves the problem of unbalanced samples in the work of the one-stage detector. Overall, the single-stage detector can improve its detection accuracy while maintaining a high detection speed. In 2018, Liu et al. put forward an efficient one-stage pedestrian detection architecture ALFnet [71], which mainly uses the continuously increasing IOU threshold to train multiple positioning modules. It can improve the detection accuracy of pedestrian detection. It can achieve the same detection speed as SSD and the same detection accuracy as Faster RCNN; at that time, the most advanced performance was achieved on the CityPersons dataset and the Caltech dataset. In 2019, Zheng et al. proposed DIOU Loss and CIOU Loss [72] to optimize the previous loss function. Compared with the previous object box regression loss, it considers the overlap area, center point distance, and aspect ratio. The bounding box considering the distance loss has faster convergence speed and higher convergence accuracy, which improves the detection accuracy of the object detection framework.In 2020, Alexey Bochkovskiy proposed YOLOv4 [73]. Based on the advantages of multiple detection frameworks, Backbone partially uses the CSPNet structure [74] proposed by Wang et al. in 2020. The schematic diagram of applying CSP to ResNet is shown in Figure 6, which adds a path to each cycle block. In the neck part, the feature fusion is performed by adding the SPP structure [75] and the PAN structure [76]. In addition, the advantages of clustering [77] are used to generate the predicted frame size. The SPP structure can help the network integrate the features of different scales, and the PAN structure integrates the features obtained from different layers. Finally, YOLOv4 obtains 65.7% (AP50) detection accuracy and 65FPS detection speed on the coco dataset, achieving the best balance between the current detection frame speed and accuracy. In addition, some scholars study the application of object detection to slam so as to promote the development of related technologies [78]. So far, pedestrian detection algorithms have mostly focused on the two-stage network framework. However, pedestrian detection in intelligent driving technology has high requirements for real-time performance. With the breakthroughs in accuracy and real-time performance of frameworks such as YOLOv4, pedestrian detection technology in the future will focus more on the single-stage detection framework. Beyond that, these algorithms have also laid a solid foundation for the application of pedestrian detection in intelligent driving.Figure 6
CSPNet structure [74].Among the above-mentioned one-stage detection methods based on deep learning, the YOLO series methods (YOLOv1 [65], YOLOv2 [67], YOLOv3 [68], and YOLOv4 [73]) are the earliest ones proposed in recent years. The algorithms of the YOLO series can be used in various object detection tasks. Due to the reason that only a limited number of objects are predicted in one anchor, it often causes missed detection in the scene of crowded pedestrians, so the performance of the algorithm will be reduced in the crowded scene. However, the high detection speed of such algorithms provides the possibility for the application of pedestrian detection technology in the field of intelligent driving. The SSD [69] algorithm mentioned above is proposed for general object detection, which can improve the problem of multiscale detection in pedestrian detection. The RetinaNet [70] detector introduces a new loss function, which can improve the detection accuracy in the general object detection field. The ALFnet [71] algorithm is mainly used for pedestrian detection. Due to the effective improvement of the task of pedestrian detection, it can be extended to general object detection to a certain extent. The CIOU Loss [72] algorithm researches the boundary regression problem in object detection, which effectively improves the detection effect of various objects. Table 2 summarizes the calculation speeds of the one-stage detection methods mentioned above.Table 2
Calculation speed of some one-stage algorithms.
MethodBackboneRate (fps)YOLOv1 [65]—45YOLOv2 [67]Darknet40YOLOv3 [68]Darknet34.5YOLOv4 [73]Darknet65SSD [69]VGG59RetinaNet [70]ResNet-FPN13.7ALFnet [71]ResNet3.7


## 2.3. Backbone

The pedestrian detection algorithms are different. However, in the deep learning stage, the first is to use the convolutional neural network to process the image to obtain the deep feature map and then perform various subsequent processing. This part obtains the convolutional neural network of the feature map called the “Backbone” of the entire algorithm. Backbone can decisively influence the effect of the network. This section will review this content.


### 2.3.1. VGGNet

After AlexNet [79] achieves excellent results in the ImageNet competition, the VGGNet [80] proposed by Simonyan in 2014 improves the convolutional neural network, uses a smaller convolution kernel and a deeper network structure, and achieves better results.


### 2.3.2. INception

In the process of extracting features of the convolutional neural network, increasing the depth and width of the network can improve the performance of the network. Nonetheless, doing so will also lead to a substantial increase in the number of parameters, and it is prone to overfitting. Inception [81], proposed in 2014, solves this problem better. It uses three convolution kernels of different sizes for convolution calculations and then cascades these parts to enter the next layer. Later, the improved v2, v3, and v4 versions [82–84] are proposed.


### 2.3.3. ResNet

Based on VGGNet and Inception, He et al. proposed ResNet [85] in 2015, solving the problem of gradient disappearance and gradient update difficulty. Since then, ResNet has been generally used as Backbone for various classification, detection, and segmentation tasks. The main idea is to introduce a residual block, let the convolutional network learn the residual mapping, and make the network optimization easier.


### 2.3.4. DenseNet

In 2017, DenseNet [86] maximized the information exchange between the front and rear layers based on ResNet. By establishing dense connections between all the front layers and all the back layers, it realizes the multiplexing of features in the channel dimension. This structure can achieve better performance than ResNet with fewer parameters and calculations.


### 2.3.5. FPN

In order to enhance semantics, traditional object detection models usually only perform follow-up operations on the last feature layer, but the final feature map often has less detailed information, making the detection of small objects more difficult. In 2017, the FPN method merged the features of different layers, which better improves the multiscale detection problem. The overall architecture of FPN mainly consists of four parts: bottom-up network, top-down network, horizontal connection, and convolution.


### 2.3.6. DetNet

DetNet [87] introduces the hole convolution, which increases the receptive field, obtains a larger feature map size, and makes the model have a larger receptive field and higher resolution. At the same time, the detection of large objects and small objects is taken into account. It is especially suitable for inspection tasks. The structure diagram is shown in Figure 7.Figure 7
DetNet structure diagram [87]. (a) Dilated bottleneck. (b) Dilated bottleneck with 1 × 1 conv projection. (c) Original bottleneck.
(a)(b)(c)


## 2.3.1. VGGNet

After AlexNet [79] achieves excellent results in the ImageNet competition, the VGGNet [80] proposed by Simonyan in 2014 improves the convolutional neural network, uses a smaller convolution kernel and a deeper network structure, and achieves better results.


## 2.3.2. INception

In the process of extracting features of the convolutional neural network, increasing the depth and width of the network can improve the performance of the network. Nonetheless, doing so will also lead to a substantial increase in the number of parameters, and it is prone to overfitting. Inception [81], proposed in 2014, solves this problem better. It uses three convolution kernels of different sizes for convolution calculations and then cascades these parts to enter the next layer. Later, the improved v2, v3, and v4 versions [82–84] are proposed.


## 2.3.3. ResNet

Based on VGGNet and Inception, He et al. proposed ResNet [85] in 2015, solving the problem of gradient disappearance and gradient update difficulty. Since then, ResNet has been generally used as Backbone for various classification, detection, and segmentation tasks. The main idea is to introduce a residual block, let the convolutional network learn the residual mapping, and make the network optimization easier.


## 2.3.4. DenseNet

In 2017, DenseNet [86] maximized the information exchange between the front and rear layers based on ResNet. By establishing dense connections between all the front layers and all the back layers, it realizes the multiplexing of features in the channel dimension. This structure can achieve better performance than ResNet with fewer parameters and calculations.


## 2.3.5. FPN

In order to enhance semantics, traditional object detection models usually only perform follow-up operations on the last feature layer, but the final feature map often has less detailed information, making the detection of small objects more difficult. In 2017, the FPN method merged the features of different layers, which better improves the multiscale detection problem. The overall architecture of FPN mainly consists of four parts: bottom-up network, top-down network, horizontal connection, and convolution.


## 2.3.6. DetNet

DetNet [87] introduces the hole convolution, which increases the receptive field, obtains a larger feature map size, and makes the model have a larger receptive field and higher resolution. At the same time, the detection of large objects and small objects is taken into account. It is especially suitable for inspection tasks. The structure diagram is shown in Figure 7.Figure 7
DetNet structure diagram [87]. (a) Dilated bottleneck. (b) Dilated bottleneck with 1 × 1 conv projection. (c) Original bottleneck.
(a)(b)(c)


## 3. Dataset and Evaluation Method

### 3.1. Dataset

The dataset is the basis of the pedestrian detection task. It not only is a data source for researchers to conduct experimental tests but also provides the same data basis for the performance comparison of different algorithms. Measuring the quality of a dataset includes the amount of data and the quality of labeled information. The richness of the dataset determines the robustness of the detector to a certain extent. Compared with general object detection tasks, pedestrian detection has its own unique characteristics. Common pedestrian detection datasets now include Caltech [88], KITTI [89], CityPersons [90], TUD [91], and EuroCity [92]. In addition, the current common dataset in the object detection field is COCO. The relevant information of these datasets is shown in Table 3. According to the different content of each dataset, it has its own characteristics. Among them, the Caltech, KITTI, and CityPersons datasets have more complete labeling information and are more widely used. The images in these three datasets are shown in Figure 8. Here is a brief introduction to these datasets.Table 3
Dataset related information.
Dataset nameCategoryImages of trainImages of testSizeCharacteristicCaltech1192000 (persons)155000 (persons)640 × 480Large amount of data and rich annotation informationKITTI8748175181242 × 375Capture datasets in rural areas and highways, each image contains up to 15 cars and 30 pedestriansCityPersons129755002048 × 1024The training set contains approximately 19,744 pedestrians and the test set contains 11 000 pedestriansEuroCity147300 (238300 persons)—1920 × 1024Pedestrians and riders are carefully marked; especially posters and portraits are marked separatelyTUD11284250720 × 576Evaluate the role of motion information in pedestrian detection and provide image pairs to calculate optical flow informationCOCO8011800046000—Multiple categories, large-scale datasetsFigure 8
Some example images in (a) Caltech [88], (b) KITTI [89], and (c) CityPersons [90].
(a)(b)(c)


#### 3.1.1. CALTECH

Caltech is currently the largest pedestrian detection dataset, which includes 350,000 pedestrian bounding boxes marked in 250,000 frames of images, and the occlusion and the corresponding time are also marked.


#### 3.1.2. KITTI

The KITTI dataset is currently the largest computer vision algorithm evaluation dataset in autonomous driving scenarios. This dataset is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection, and 3D tracking in a vehicle environment. KITTI contains real image data collected from scenes such as urban, villages, and highways. There are up to 15 cars and 30 pedestrians in each image, with various degrees of occlusion and truncation.


#### 3.1.3. CityPersons

The Cityscapes city dataset contains street scenes from 50 different cities recorded from a set of different stereo video sequences and the pixel-level annotation of the image. It mainly labels the data of pedestrians on urban roads to obtain a pedestrian detection dataset.


### 3.2. Evaluation Method

The detection ability of the pedestrian detector is mainly reflected by the corresponding evaluation index, and an excellent evaluation method can objectively reflect the detection ability of the detector. Generally, the detector is trained through the train set of the dataset, and then the detector is evaluated through the test set.At present, the most commonly used evaluation for object detection is Average Precision (AP). Generally, the performance of the model is dynamically evaluated by drawing a P-R curve, where the horizontal coordinate is the recall rate and the vertical coordinate is the accuracy rate. In order to compare the performance of all object categories in multiclass detection, the mean Average Precision (mAP) of all object categories is usually used as the final metric of performance. In order to measure the accuracy of object positioning, Intersection over Union (IoU) is used to check whether the overlap ratio between the prediction box and the ground truth box is greater than a predefined threshold, which is generally defined as 0.5. If it is greater than this value, the object will be recognized as successfully detected; otherwise, it will be defined as missed. After 2014, due to the widespread use of COCO datasets, researchers began to pay more attention to accuracy. In COCO, a fixed IoU threshold is not used. Instead, take the average of multiple IoU thresholds between 0.5 (coarse positioning) and 0.95 (perfect positioning). This metric change promotes more accurate object positioning.In addition, some scholars found in their research that only using the precision-recall curve cannot accurately express the effectiveness. Piotr proposed the MR-FPPI curve in 2012, where MR represents the missed detection rate and FPPI represents the number of false detections per image. This evaluation method is commonly used in the field of pedestrian detection.In the Caltech dataset, the detection results of some of the most advanced algorithms for pedestrian detection in overall data, far scale data, and heavy occlusion data are shown in Tables4–6.Table 4
Overall data.
MethodsMiss rate (%)MethodsMiss rate (%)SA-FastRCNN [64]63F-DNN51PCN [93]62F-DNN + SS50SDS-RCNN [94]61F-DNN2 + SS50MS-CNN [95]61GDFL48AdaptFasterRCNN [90]60ADM42AR-Ped [96]59TLL-TFA38FasterRCNN + ATT [97]55Table 5
Far scale data.
MethodsMiss rate (%)MethodsMiss rate (%)MultiFtr + CSS97F-DNN77CrossTalk97F-dnn + SS77AFS + Geo97F-DNN2 + SS76FeatSynth96ADM75FPDW96GDFL71ChnFtrs95TLL-TFA60FasterRCNN + ATT91Table 6
Heavy occlusion data.
MethodsMiss rate (%)MethodsMiss rate (%)DeepParts60AR-Ped49MS-CNN60FasterRCNN + ATT45SDS-RCNN59GDFL43AdaptFasterRCNN58F-DNN2 + SS40PCN56ADM30F-DNN55TLL-TFA29F-dnn + SS54


## 3.1. Dataset

The dataset is the basis of the pedestrian detection task. It not only is a data source for researchers to conduct experimental tests but also provides the same data basis for the performance comparison of different algorithms. Measuring the quality of a dataset includes the amount of data and the quality of labeled information. The richness of the dataset determines the robustness of the detector to a certain extent. Compared with general object detection tasks, pedestrian detection has its own unique characteristics. Common pedestrian detection datasets now include Caltech [88], KITTI [89], CityPersons [90], TUD [91], and EuroCity [92]. In addition, the current common dataset in the object detection field is COCO. The relevant information of these datasets is shown in Table 3. According to the different content of each dataset, it has its own characteristics. Among them, the Caltech, KITTI, and CityPersons datasets have more complete labeling information and are more widely used. The images in these three datasets are shown in Figure 8. Here is a brief introduction to these datasets.Table 3
Dataset related information.
Dataset nameCategoryImages of trainImages of testSizeCharacteristicCaltech1192000 (persons)155000 (persons)640 × 480Large amount of data and rich annotation informationKITTI8748175181242 × 375Capture datasets in rural areas and highways, each image contains up to 15 cars and 30 pedestriansCityPersons129755002048 × 1024The training set contains approximately 19,744 pedestrians and the test set contains 11 000 pedestriansEuroCity147300 (238300 persons)—1920 × 1024Pedestrians and riders are carefully marked; especially posters and portraits are marked separatelyTUD11284250720 × 576Evaluate the role of motion information in pedestrian detection and provide image pairs to calculate optical flow informationCOCO8011800046000—Multiple categories, large-scale datasetsFigure 8
Some example images in (a) Caltech [88], (b) KITTI [89], and (c) CityPersons [90].
(a)(b)(c)


### 3.1.1. CALTECH

Caltech is currently the largest pedestrian detection dataset, which includes 350,000 pedestrian bounding boxes marked in 250,000 frames of images, and the occlusion and the corresponding time are also marked.


### 3.1.2. KITTI

The KITTI dataset is currently the largest computer vision algorithm evaluation dataset in autonomous driving scenarios. This dataset is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection, and 3D tracking in a vehicle environment. KITTI contains real image data collected from scenes such as urban, villages, and highways. There are up to 15 cars and 30 pedestrians in each image, with various degrees of occlusion and truncation.


### 3.1.3. CityPersons

The Cityscapes city dataset contains street scenes from 50 different cities recorded from a set of different stereo video sequences and the pixel-level annotation of the image. It mainly labels the data of pedestrians on urban roads to obtain a pedestrian detection dataset.


## 3.1.1. CALTECH

Caltech is currently the largest pedestrian detection dataset, which includes 350,000 pedestrian bounding boxes marked in 250,000 frames of images, and the occlusion and the corresponding time are also marked.


## 3.1.2. KITTI

The KITTI dataset is currently the largest computer vision algorithm evaluation dataset in autonomous driving scenarios. This dataset is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection, and 3D tracking in a vehicle environment. KITTI contains real image data collected from scenes such as urban, villages, and highways. There are up to 15 cars and 30 pedestrians in each image, with various degrees of occlusion and truncation.


## 3.1.3. CityPersons

The Cityscapes city dataset contains street scenes from 50 different cities recorded from a set of different stereo video sequences and the pixel-level annotation of the image. It mainly labels the data of pedestrians on urban roads to obtain a pedestrian detection dataset.


## 3.2. Evaluation Method

The detection ability of the pedestrian detector is mainly reflected by the corresponding evaluation index, and an excellent evaluation method can objectively reflect the detection ability of the detector. Generally, the detector is trained through the train set of the dataset, and then the detector is evaluated through the test set.At present, the most commonly used evaluation for object detection is Average Precision (AP). Generally, the performance of the model is dynamically evaluated by drawing a P-R curve, where the horizontal coordinate is the recall rate and the vertical coordinate is the accuracy rate. In order to compare the performance of all object categories in multiclass detection, the mean Average Precision (mAP) of all object categories is usually used as the final metric of performance. In order to measure the accuracy of object positioning, Intersection over Union (IoU) is used to check whether the overlap ratio between the prediction box and the ground truth box is greater than a predefined threshold, which is generally defined as 0.5. If it is greater than this value, the object will be recognized as successfully detected; otherwise, it will be defined as missed. After 2014, due to the widespread use of COCO datasets, researchers began to pay more attention to accuracy. In COCO, a fixed IoU threshold is not used. Instead, take the average of multiple IoU thresholds between 0.5 (coarse positioning) and 0.95 (perfect positioning). This metric change promotes more accurate object positioning.In addition, some scholars found in their research that only using the precision-recall curve cannot accurately express the effectiveness. Piotr proposed the MR-FPPI curve in 2012, where MR represents the missed detection rate and FPPI represents the number of false detections per image. This evaluation method is commonly used in the field of pedestrian detection.In the Caltech dataset, the detection results of some of the most advanced algorithms for pedestrian detection in overall data, far scale data, and heavy occlusion data are shown in Tables4–6.Table 4
Overall data.
MethodsMiss rate (%)MethodsMiss rate (%)SA-FastRCNN [64]63F-DNN51PCN [93]62F-DNN + SS50SDS-RCNN [94]61F-DNN2 + SS50MS-CNN [95]61GDFL48AdaptFasterRCNN [90]60ADM42AR-Ped [96]59TLL-TFA38FasterRCNN + ATT [97]55Table 5
Far scale data.
MethodsMiss rate (%)MethodsMiss rate (%)MultiFtr + CSS97F-DNN77CrossTalk97F-dnn + SS77AFS + Geo97F-DNN2 + SS76FeatSynth96ADM75FPDW96GDFL71ChnFtrs95TLL-TFA60FasterRCNN + ATT91Table 6
Heavy occlusion data.
MethodsMiss rate (%)MethodsMiss rate (%)DeepParts60AR-Ped49MS-CNN60FasterRCNN + ATT45SDS-RCNN59GDFL43AdaptFasterRCNN58F-DNN2 + SS40PCN56ADM30F-DNN55TLL-TFA29F-dnn + SS54


## 4. General Issues

At present, the detection ability of mainstream detectors for general images has been developed by leaps and bounds, especially the images of short distances and large objects for which very good detection results can be obtained. Currently, the main restriction of the further development of pedestrian detection lies in the detection ability for low-quality images, including the key issues such as multiscale and occlusion. This section will analyze these issues.


### 4.1. Occlusion Issue

Crowding and occlusion between objects are the common difficulties in pedestrian detection [98], as shown in Figure 9, causing the loss of information of the object, and invisibility of part of the area, which is likely to cause false or missed detection by the detector.Figure 9
Congestion and obscuration of pedestrian images.Compared with general object detection, occlusion is more likely to happen in pedestrian detection because group movement behaviors are prone to occur in pedestrians, which is also a major obstacle limiting the application of pedestrian detection in autonomous driving tasks. In the CityPersons dataset, the proportion of pedestrian occlusion is shown in Table7, and the occlusion between pedestrians has a serious impact on the accuracy of pedestrian positioning, which is more sensitive to the NMS threshold, thereby easily suppressing the candidate frames of similar pedestrians.Table 7
Pedestrian occlusion ratio in the CityPersons dataset.
Total number of pedestriansIoU between pedestrians is greater than 0.1 (%)IoU between pedestrians is greater than 0.3 (%)315748.826.4Due to the lack of information for pedestrians under occlusion, researchers used variable part models to solve the related problems at the beginning. Although the detection results have been improved to a certain extent, the amount of model calculations has increased sharply [99–101]. To break through the limitations of multicomponent detectors, Ouyang et al. integrated detectors with occlusion of different degrees [102], thus effectively shortening the detection time, and in the further research integrated the part model into the neural network to improve the detection effect. Though an effective method is available to improve the effect of pedestrian detection under occlusion based on partial model-assisted global detection [103], it is at the price of increased computational cost and reduced detection speed. Therefore, one of the main research directions of this method is to improve the recognition rate of the detector for blocked pedestrians while maintaining the detection speed.Similar to the part model method that uses a series of component detectors to merge with each other, another solution takes the advantages of the attention mechanism [104] to focus on key parts of pedestrians for occlusion detection. As a model, SSA-CNN [105] uses the attention mechanism to perform occlusion detection, thereby effectively improving the detection effect. In addition, some methods such as SDS-RCNN use semantic segmentation to deal with the occlusion problems, in this manner to make the generated features more focused on pedestrians, locate possible pedestrian areas, and have CNN paying attention to possible pedestrian occlusion parts. The main idea of this method is to quickly locate pedestrians and focus on the characteristics of the pedestrian’s location. The SDS-RCNN framework is shown in Figure 10.Figure 10
Framework diagram of SDS-RCNN [94].In addition to the above-mentioned methods used to solve the occlusion problem in pedestrian detection, some scholars focused on postprocessing. Liu et al. proposed an Adaptive NMS [106] method to solve the problem of sensitivity to the NMS threshold in pedestrian detection, thereby effectively improving the detection efficiency. In addition, Wang et al. designed a new repulsive loss function RepLoss [107] to reduce the mutual influence between objects, which effectively alleviates the detection effect in the case of pedestrian occlusion. Zhang et al. proposed that OR-CNN [108] can improve the loss function and ROI Pooling based on Faster RCNN and introduced the idea of part-based which effectively alleviates the problem of pedestrian occlusion.At present, the processing of pedestrian detection and occlusion problems has gradually shifted to the CNN itself and from the improvement of the overall network architecture to the improvement of each processing stage.Among the algorithms introduced above, the algorithms [99–101] are all early methods based on deformable parts, which are mainly used for pedestrian detection. This type of method is not universal and needs to be designed for specific detection objects. Similarly, the algorithm [102, 103] is also designed for the problem of pedestrian occlusion, and it is difficult to generalize to the field of general object detection. SDS-RCNN [94] and SSA-CNN [105] are mainly designed for pedestrian detection to improve the effect of pedestrian detection. Adaptive NMS [106] is mainly designed for the crowding problem in pedestrian detection. This algorithm can be extended to the general object detection field to a certain extent, reducing the error of common use NMS algorithms. Similar to [106], RepLoss [107] and OR-CNN [108] are mainly designed for pedestrian detection and can be extended to general object detection to a certain extent. However, because these algorithms are specifically designed for pedestrian detection, the improvement in general object detection is limited. The calculation speeds of some of the above algorithms are summarized in Table 8.Table 8
Calculation speed of some occlusion problem processing algorithm.
MethodBackboneRate (fps)SDS-RCNN [94]VGG5SSA-CNN [105]VGG9.1Adaptive NMS [106]VGG—RepLoss [107]ResNet—OR-CNN [108]VGG16—


### 4.2. Multiscale Issue

The traditional convolutional neural network adopts a single-line structure, and the shallow feature map has a larger area and contains more detailed information, making it suitable for detecting small objects. The deep feature map, which has a small area and only contains semantic information, is suitable for the detection of large objects. Generally, convolutional neural networks present the problem of multiscale detection of large and small objects, which has not been well solved [109]. The multiscale pedestrian image is illustrated in Figure 11. For small object detection, reducing the downsampling rate of the network, which is the simplest way to improve detection capability, can increase the detailed information on the feature map. Besides, a hole convolution can be used to increase the receptive field of the subsequent layer when the downsampling rate is reduced. This convolution method cannot guarantee that the receptive field after the modification is consistent with that before the modification but can minimize the degree of change as much as possible. Moreover, many methods [110–112] have been proposed to solve this problem.Figure 11
Multiscale pedestrian image.With the purpose of improving the multiscale detection capability, several different image input scales can be set in the training phase. During training, one is randomly selected from multiple scales, and the picture is scaled to this scale and input into the network, contributing to an increase in the robustness of the network without raising the amount of calculation. Song et al. proposed the TLL method [113], which improved the detection results by establishing human body model information at different scales. However, Zhang et al. have effectively reduced the missed detection rate by further investigating the label information [114].With the increase in the number of layers, the traditional convolutional network will enlarge the receptive field and enrich the semantic information while causing severe loss of the information of the small object at the output of the network. Its small object detection ability is very poor. The idea of feature fusion [115–120] is to combine deep and shallow layers, fuse the features of the two, and complement each other’s advantages, so as to improve detection performance. RPN has this effect; however, the improvement of pedestrian detection effect for small objects is limited. Li et al. and Cai et al. proposed SAF RCNN and MS-CNN, respectively, to deal with scale changes. Besides, SSD also enhances the detection effect by combining different feature layers for feature fusion. Generally, the key to multiscale detection is whether the feature extraction stage can extract pedestrian features at various scales.The researcher proposing the TridentNet network [121] changed the number of holes in the last convolutional layer by analyzing the influence of different sizes of receptive fields on the detection results. He parallelized three different receptive fields and compared the previous basic network results. The detection results demonstrate a significant improvement in accuracy. The network diagram is illustrated in Figure 12.Figure 12
Framework diagram of TridentNet [121].In pedestrian detection, the current effective methods for solving multiscale problems include reducing the downsampling rate and convolution of holes, multiscale training MST, feature fusion, and TridentNet. The core idea is to obtain more general detection capabilities at different scales by fully excavating the feature information of different scale features.Among the algorithms introduced above, the algorithms [110–112] are all designed for pedestrian detection, and part of their content can be extended to general object detection. Similarly, the algorithms [113, 114] are also designed to solve the problem of pedestrian detection and are used to improve the performance of pedestrian detection at different scales. Both MS-CNN [95] and TridentNet [121] are designed for general object detection, and good results can also be obtained in pedestrian detection technology. Compared with other object detection tasks, object scale changes are more common in the field of pedestrian detection, so the algorithm mentioned above can effectively change the effect of pedestrian detection. The calculation speeds of some of the above algorithms are summarized in Table 9.Table 9
Calculation speed of some multiscale problem processing algorithm.
MethodBackboneRate (fps)MCF [110]VGG0.54 (CPU)HyperLearner [111]VGG7.1CFM [112]VGG0.8TLL [113]ResNet—MS-CNN [95]VGG15TridentNet [121]ResNet—


## 4.1. Occlusion Issue

Crowding and occlusion between objects are the common difficulties in pedestrian detection [98], as shown in Figure 9, causing the loss of information of the object, and invisibility of part of the area, which is likely to cause false or missed detection by the detector.Figure 9
Congestion and obscuration of pedestrian images.Compared with general object detection, occlusion is more likely to happen in pedestrian detection because group movement behaviors are prone to occur in pedestrians, which is also a major obstacle limiting the application of pedestrian detection in autonomous driving tasks. In the CityPersons dataset, the proportion of pedestrian occlusion is shown in Table7, and the occlusion between pedestrians has a serious impact on the accuracy of pedestrian positioning, which is more sensitive to the NMS threshold, thereby easily suppressing the candidate frames of similar pedestrians.Table 7
Pedestrian occlusion ratio in the CityPersons dataset.
Total number of pedestriansIoU between pedestrians is greater than 0.1 (%)IoU between pedestrians is greater than 0.3 (%)315748.826.4Due to the lack of information for pedestrians under occlusion, researchers used variable part models to solve the related problems at the beginning. Although the detection results have been improved to a certain extent, the amount of model calculations has increased sharply [99–101]. To break through the limitations of multicomponent detectors, Ouyang et al. integrated detectors with occlusion of different degrees [102], thus effectively shortening the detection time, and in the further research integrated the part model into the neural network to improve the detection effect. Though an effective method is available to improve the effect of pedestrian detection under occlusion based on partial model-assisted global detection [103], it is at the price of increased computational cost and reduced detection speed. Therefore, one of the main research directions of this method is to improve the recognition rate of the detector for blocked pedestrians while maintaining the detection speed.Similar to the part model method that uses a series of component detectors to merge with each other, another solution takes the advantages of the attention mechanism [104] to focus on key parts of pedestrians for occlusion detection. As a model, SSA-CNN [105] uses the attention mechanism to perform occlusion detection, thereby effectively improving the detection effect. In addition, some methods such as SDS-RCNN use semantic segmentation to deal with the occlusion problems, in this manner to make the generated features more focused on pedestrians, locate possible pedestrian areas, and have CNN paying attention to possible pedestrian occlusion parts. The main idea of this method is to quickly locate pedestrians and focus on the characteristics of the pedestrian’s location. The SDS-RCNN framework is shown in Figure 10.Figure 10
Framework diagram of SDS-RCNN [94].In addition to the above-mentioned methods used to solve the occlusion problem in pedestrian detection, some scholars focused on postprocessing. Liu et al. proposed an Adaptive NMS [106] method to solve the problem of sensitivity to the NMS threshold in pedestrian detection, thereby effectively improving the detection efficiency. In addition, Wang et al. designed a new repulsive loss function RepLoss [107] to reduce the mutual influence between objects, which effectively alleviates the detection effect in the case of pedestrian occlusion. Zhang et al. proposed that OR-CNN [108] can improve the loss function and ROI Pooling based on Faster RCNN and introduced the idea of part-based which effectively alleviates the problem of pedestrian occlusion.At present, the processing of pedestrian detection and occlusion problems has gradually shifted to the CNN itself and from the improvement of the overall network architecture to the improvement of each processing stage.Among the algorithms introduced above, the algorithms [99–101] are all early methods based on deformable parts, which are mainly used for pedestrian detection. This type of method is not universal and needs to be designed for specific detection objects. Similarly, the algorithm [102, 103] is also designed for the problem of pedestrian occlusion, and it is difficult to generalize to the field of general object detection. SDS-RCNN [94] and SSA-CNN [105] are mainly designed for pedestrian detection to improve the effect of pedestrian detection. Adaptive NMS [106] is mainly designed for the crowding problem in pedestrian detection. This algorithm can be extended to the general object detection field to a certain extent, reducing the error of common use NMS algorithms. Similar to [106], RepLoss [107] and OR-CNN [108] are mainly designed for pedestrian detection and can be extended to general object detection to a certain extent. However, because these algorithms are specifically designed for pedestrian detection, the improvement in general object detection is limited. The calculation speeds of some of the above algorithms are summarized in Table 8.Table 8
Calculation speed of some occlusion problem processing algorithm.
MethodBackboneRate (fps)SDS-RCNN [94]VGG5SSA-CNN [105]VGG9.1Adaptive NMS [106]VGG—RepLoss [107]ResNet—OR-CNN [108]VGG16—


## 4.2. Multiscale Issue

The traditional convolutional neural network adopts a single-line structure, and the shallow feature map has a larger area and contains more detailed information, making it suitable for detecting small objects. The deep feature map, which has a small area and only contains semantic information, is suitable for the detection of large objects. Generally, convolutional neural networks present the problem of multiscale detection of large and small objects, which has not been well solved [109]. The multiscale pedestrian image is illustrated in Figure 11. For small object detection, reducing the downsampling rate of the network, which is the simplest way to improve detection capability, can increase the detailed information on the feature map. Besides, a hole convolution can be used to increase the receptive field of the subsequent layer when the downsampling rate is reduced. This convolution method cannot guarantee that the receptive field after the modification is consistent with that before the modification but can minimize the degree of change as much as possible. Moreover, many methods [110–112] have been proposed to solve this problem.Figure 11
Multiscale pedestrian image.With the purpose of improving the multiscale detection capability, several different image input scales can be set in the training phase. During training, one is randomly selected from multiple scales, and the picture is scaled to this scale and input into the network, contributing to an increase in the robustness of the network without raising the amount of calculation. Song et al. proposed the TLL method [113], which improved the detection results by establishing human body model information at different scales. However, Zhang et al. have effectively reduced the missed detection rate by further investigating the label information [114].With the increase in the number of layers, the traditional convolutional network will enlarge the receptive field and enrich the semantic information while causing severe loss of the information of the small object at the output of the network. Its small object detection ability is very poor. The idea of feature fusion [115–120] is to combine deep and shallow layers, fuse the features of the two, and complement each other’s advantages, so as to improve detection performance. RPN has this effect; however, the improvement of pedestrian detection effect for small objects is limited. Li et al. and Cai et al. proposed SAF RCNN and MS-CNN, respectively, to deal with scale changes. Besides, SSD also enhances the detection effect by combining different feature layers for feature fusion. Generally, the key to multiscale detection is whether the feature extraction stage can extract pedestrian features at various scales.The researcher proposing the TridentNet network [121] changed the number of holes in the last convolutional layer by analyzing the influence of different sizes of receptive fields on the detection results. He parallelized three different receptive fields and compared the previous basic network results. The detection results demonstrate a significant improvement in accuracy. The network diagram is illustrated in Figure 12.Figure 12
Framework diagram of TridentNet [121].In pedestrian detection, the current effective methods for solving multiscale problems include reducing the downsampling rate and convolution of holes, multiscale training MST, feature fusion, and TridentNet. The core idea is to obtain more general detection capabilities at different scales by fully excavating the feature information of different scale features.Among the algorithms introduced above, the algorithms [110–112] are all designed for pedestrian detection, and part of their content can be extended to general object detection. Similarly, the algorithms [113, 114] are also designed to solve the problem of pedestrian detection and are used to improve the performance of pedestrian detection at different scales. Both MS-CNN [95] and TridentNet [121] are designed for general object detection, and good results can also be obtained in pedestrian detection technology. Compared with other object detection tasks, object scale changes are more common in the field of pedestrian detection, so the algorithm mentioned above can effectively change the effect of pedestrian detection. The calculation speeds of some of the above algorithms are summarized in Table 9.Table 9
Calculation speed of some multiscale problem processing algorithm.
MethodBackboneRate (fps)MCF [110]VGG0.54 (CPU)HyperLearner [111]VGG7.1CFM [112]VGG0.8TLL [113]ResNet—MS-CNN [95]VGG15TridentNet [121]ResNet—


## 5. Discussion

Object detection is one of the four basic tasks of computer vision, and it is a current research hotspot. The main purpose of this task is to detect specific object instances (“cats,” “dogs,” etc.) from a given image. As a typical object detection task, pedestrian detection is consistent with the general object detection task, which is to detect pedestrians in a given image. In recent years, with the continuous development of deep learning [122, 123], object detection has made great progress, especially the wide application of multicategory datasets such as COCO. Most researches focus on general object detection. As a typical object detection task, pedestrian detection has a special position in fields such as intelligent driving, and it is directly related to driving safety and pedestrian safety. At present, due to the widespread attention of general object detection, there are few reviews in the field of pedestrian detection. For example, references [24, 27] gave a full introduction to general object detection in recent years but did not conduct a detailed analysis of pedestrian detection. Reference [124] mainly discusses the problem of pedestrian detection in far-infrared video and does not involve pedestrian detection technology in natural images. References [88, 114, 125, 126] did not discuss the research progress in the past two years due to time constraints and rarely involved the current research focus on deep learning techniques. Reference [127] mainly discusses Human Detection technology and does not make detailed analysis for pedestrian detection.Based on the general analysis of general object detection, this paper makes an in-depth discussion on pedestrian detection problems. The main contributions of this paper are as follows: (1) The pedestrian detection algorithm based on deep learning proposed in recent years is introduced in detail, and its advantages and disadvantages are analyzed. (2) It introduces the common use datasets and evaluation metrics for pedestrian detection. (3) The main issues that limit the performance of pedestrian detection in areas such as intelligent driving are discussed in detail. (4) It explains the future development direction of pedestrian detection. However, this paper does not involve the introduction of pedestrian detection in special scenarios (night, rain, snow, fog, etc.), which is also the direction of future work.The pedestrian detection technology described in this review is mainly solved by visual methods based on machine learning technology, which is also the current mainstream solution. However, this solution has certain constraints. Although vision-based image processing technology has made great progress, this method has a higher demand for the external environment (light, weather, etc.). On this basis, some people have paid attention to the research of infrared images and made some progress. However, the lack of infrared image datasets limits its development to a certain extent, and it is still sensitive to factors such as occlusion. Vision-based detection technology has its inherent constraints. How to use multisensor fusion technology to improve the effect of pedestrian detection technology in practical applications such as intelligent driving is a major development direction at present. In addition, although the traditional machine learning technology has fast detection speed and low hardware platform requirements, it can no longer meet the current application requirements due to its low detection accuracy. Although the deep learning technology in machine learning technology has made great progress in recent years, the computing model is often large and has high requirements on the hardware platform. It is more difficult to deploy on the mobile terminal with less computing resources such as smart cars. This is also a major factor affecting the development of deep learning technology.


## 6. Conclusions

Pedestrian detection is an important problem of computer vision. Compared with general object detection, it has important research value in the field of intelligent driving. It has similarities and differences with general object detection. This review first introduces the content of general object detection, then analyzes the development of pedestrian detection, and elaborates on the common datasets and main problems faced by pedestrian detection. Although the pedestrian detection technology has made great progress from the original traditional machine learning to the current neural network, there is still a huge gap with human vision. In addition, lightweight network is also a research core. How to deploy it to the mobile terminal without affecting performance directly affects its application in intelligent driving. This review believes that the future development direction of pedestrian detection technology is as follows:(1)
The above-mentioned multiscale issues and occlusion issues are the core issues affecting pedestrian detection. Among them, the multiscale issue requires that pedestrians of different sizes can be accurately detected at the same time, which puts higher requirements on the feature extraction network. The occlusion issue requires accurate detection of pedestrian parts and puts forward higher requirements on the recognition algorithm. The improvement of these issues can directly improve the effect of pedestrian detection in complex scenes, which is an important way to improve the ability of pedestrian detectors.(2)
Although the current detection network has made great progress, the hardware requirements are often high. Therefore, how to lightweight the network while maintaining the detection performance is an important issue in practical applications. This is also an important direction for future development.(3)
At present, the general pedestrian detection still uses a single pedestrian as the object and does not consider the relationship with other objects in the environment. Considering the relationship between objects is beneficial to enhance the understanding of the scene, thereby enhancing the semantics of detection, and bringing it closer to the way of human thinking, it is an important development direction in the future.(4)
Pedestrian detection is a core technical problem in the intelligent driving. The current main solution is to use image information for detection. How to use other sensors such as lidar in intelligent driving to enhance the effect of pedestrian detection is an important research direction in the future.



---

*Source: 5410049-2021-07-21.xml*