Multi‐receptive field graph convolutional neural networks for pedestrian detection

IET Intelligent Transport SystemsVolume 13, Issue 9 p. 1319-1328 Research ArticleFree Access Multi-receptive field graph convolutional neural networks for pedestrian detection Chao Shen, Chao Shen School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorXiangmo Zhao, Xiangmo Zhao School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorXing Fan, Xing Fan School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorXinyu Lian, Xinyu Lian School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorFan Zhang, Fan Zhang School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorAbdul Rahman Kreidieh, Abdul Rahman Kreidieh Institute of Artificial Intelligence and Robotics, Xi'an JiaoTong University, Xi'an, 710049 Shaanxi, People's Republic of ChinaSearch for more papers by this authorZhanwen Liu, Corresponding Author Zhanwen Liu zwliu@chd.edu.cn School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of China Institute of Artificial Intelligence and Robotics, Xi'an JiaoTong University, Xi'an, 710049 Shaanxi, People's Republic of China Institute of Transportation Studies, University of California, Berkeley, CA, 94720 USASearch for more papers by this author Chao Shen, Chao Shen School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorXiangmo Zhao, Xiangmo Zhao School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorXing Fan, Xing Fan School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorXinyu Lian, Xinyu Lian School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorFan Zhang, Fan Zhang School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of ChinaSearch for more papers by this authorAbdul Rahman Kreidieh, Abdul Rahman Kreidieh Institute of Artificial Intelligence and Robotics, Xi'an JiaoTong University, Xi'an, 710049 Shaanxi, People's Republic of ChinaSearch for more papers by this authorZhanwen Liu, Corresponding Author Zhanwen Liu zwliu@chd.edu.cn School of Information Engineering, Chang'an University, Xi'an, 710064 Shaanxi, People's Republic of China Institute of Artificial Intelligence and Robotics, Xi'an JiaoTong University, Xi'an, 710049 Shaanxi, People's Republic of China Institute of Transportation Studies, University of California, Berkeley, CA, 94720 USASearch for more papers by this author First published: 20 May 2019 https://doi.org/10.1049/iet-its.2018.5618Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Although great progress has been achieved for general object detection with the advent of deep learning, there are still several limitations when applied to pedestrian detection task. In this article, the authors mainly consider the following three problems. (i) most deep learning-based methods are time-consuming and require large storage space, making them unsuitable for real-world applications; (ii) the scale of person varies greatly, which decreases the pedestrian detection performance; (iii) as pedestrians may be occluded by vehicles or other persons, it makes pedestrian detection more challenging. In response to these limitations, this article proposes a multi-receptive field-based framework for single-shot pedestrian detection. Authors’ contributions can be summarised as follows. First, to reduce the computational complexity of the detector, the authors design the multi-receptive pooling pyramid module, which reduces the computation cost of the detector and improves its performance. Next, body parts graphs are built on the learned convolutional neural networks (CNN) features, and a graph CNN is used to mine the relationships of the graph nodes. Finally, experiments are conducted on two public pedestrian detection datasets to demonstrate the effectiveness of the proposed method. 1 Introduction Object detection is one of the most fundamental pillars of computer vision and has been investigated for decades. Various image processing and machine learning-based methods have been proposed to improve the object detection performance. At the early stage, hand-crafted features and machine learning-based classifiers are utilised to detect objects. For example, the Viola-Jones detector [1] extracts Haar-like features and trains a cascade AdaBoost classifier to detect faces in a real-time manner. Integral channel feature (ICF) detector [2]- and aggregated channel features (ACF) [3]-based methods are proposed for more robust feature learning, which improve the detection performance of rigid objects. In order to detect non-rigid object such as people or animals, deformable part model (DPM) [4] use part filters and root filters together for better detection performance. Although these methods show promising results, it is still difficult to use them in real-time systems considering their computational cost. More recently, significant progress has been made in image classification tasks with the resurgence of deep learning methods. With the advent of large scale image classification datasets such as ImageNet [5] and more powerful GPUs (graphics processing units), deep convolutional neural networks (CNNs) such as AlexNet [6], ResNet [7], and DenseNet [8] have improved classification accuracies dramatically. By stacking convolution and pooling layers, deep CNNs can extract more abstract and high-semantic level features. As learning more discriminative and representative features is critical for many computer vision tasks, deep learning methods also promote the performance of object detection. As higher semantic level features are important to classify and locate objects in images, deep learning-based object detection frameworks have become the mainstream methods and have radically improved the performance compared to traditional methods. Deep CNN-based object detection frameworks can be divided into two categories. One is two-stage detection based on the region proposal, and another is single-shot detectors. R-CNN [9] uses selective search algorithm to generate proposals for subsequent classification and localisation. Fast R-CNN [10] uses softmax to classify regions instead of support vector machines (SVM) classifier [11]. Faster RCNN [12] proposes the region proposal networks (RPN) for faster and end-to-end detector training. Although these two-stage deep detectors can achieve higher detection performance than single-shot detectors, they are more time-consuming and difficult to use in real-time applications. YOLO [13] is a single shot object detector, which uses the final convolution feature maps to locate and classify objects. Single shot multibox detector (SSD) utilises multi-resolution feature maps to improve detection performance and is more promising to be used in real-time applications such as pedestrian detection. Pedestrian detection is an important component of Intelligent Transportation Systems, which can be used to inform drivers on the location of individuals on the road for safer driving. Although these deep CNN-based methods obtain great performance improvement for general object detection, there are still some limitations when applied to pedestrian detection tasks. One limitation is that the scale of people can vary dramatically within a frame, as shown in Fig. 1. This makes it difficult to detect accurately in real-time. Moreover, occlusion is another limitation for pedestrian detection. As accurately detecting pedestrian has great potential in various applications such as autonomous driving and surveillance, it has been studied extensively and considerable methods have been proposed to handle the scale variety and occlusion problems. Fig. 1Open in figure viewerPowerPoint Multi-Scale pedestrians (left) and occluded pedestrian (right) In response to the aforementioned pedestrian scaling concern, many multi-scale methods have been proposed to handle scale variety. The authors of [14] introduced a multiple subnetworks-based framework to detect pedestrians with scales from disjoint range. In addition, [15] demonstrated a pedestrian detection framework with scale-aware localisation policy, and it worked particularly well with far-scale pedestrians. Although these methods can promote the detection performance, they also increase the computational complexity. For real-time applications, it is critical to design low latency detection framework to handle the scale variety problem of pedestrian detection. As pointed out by [16], another main challenge of pedestrian detection is occlusion. In real world scenes such as streets, people always walk or stand together and may be occluded by vehicles or other pedestrians in the high probability. Moreover, crowded vehicles, trees, traffic signs, or traffic lights may also occlude pedestrians. Extensive methods have been proposed to handle the occlusion problem, the most common one is to learn a set of special pattern detectors to detect the occluded parts. The final detection result is obtained by the outputs of these ensembles. The drawbacks of those methods are two-fold. One is that these occlusion pattern detectors are learned independently, the final detection results may be suboptimal. Another one is that these methods are always time-consuming, and not suitable for real-time systems. Motivated by the problems depicted above, in this work, we propose a multi-receptive pooling pyramid (MRPP) module-based single-shot detector, which extracts multi-resolution and multi-receptive field features by constructing the feature pyramid with max pooling layers. To handle the occlusion problem for the single-shot detector, a body part graph is built on the final feature maps of the proposed MRPP module. Then the graph convolutional network (GCN) is used to mine the body part relationships and pass the features through graph nodes. The whole proposed network is single-shot and can be trained in an end-to-end manner, which improves the pedestrian performance and is still time-efficient. To sum up, we make the following contributions. (i) The MRPP module is proposed based on the single shot detector, which extracts the multi-resolution by constructing the pooling pyramid. Moreover, multi-receptive field features are learned with multi-branch dilated convolution layer. (ii) The GCN-based occlusion handling module on single-shot detector is proposed to improve the detection performance of occluded pedestrians. (iii) Experiments on two public datasets have shown the effectiveness of the proposed method, and the results have been comprehensively analysed. The remainder of this article is organised as follows. Section 2 presents the related work of pedestrian detection. Section 3 gives a detail description of the proposed method. The experimental results are given in Section 4. Finally, we conclude our work in Section 5. 2 Related work In this section, we will review the pedestrian detection methods from three aspects. (i) Hand-crafted models; (ii) CNN-based methods; and (iii) occlusion handling methods. 2.1 Hand-crafted models Hand-crafted feature-based models were the mainstream methods at the early stage of computer vision research. Hand-crafted models have been widely applied to object detection task. Integral images are used to speed up the feature extraction in the ICF [2] and ACF [3] methods. By training cascade AdaBoost classifiers, ICF- and ACF-based methods learn to extract important features efficiently for object detection. Histograms of oriented gradients [17] and local binary pattern [18] are also widely used in pedestrian detection, face detection, and other rigid-object detection tasks. Locally decorrelated channel features [19] were proposed to remove correlations in local image neighbourhoods to improve the detection performance. [20] introduced a multi-order context representation to exploit co-occurrence contexts of different objects. To handle the scale variety problem, some methods try to learn scale-invariant features. The work of [21] attempted to map different-resolution pedestrians to a common subspace to learn the scale-invariant features of pedestrians. Then a shared detector is trained on the mapped features to detect pedestrians. 2.2 CNN-based methods As previously mentioned, deep learning-based pedestrian detection methods have superiority in discriminative and representative feature learning, and can obtain better detection performance than traditional methods. Thus CNNs have achieved great progress in generic object detection task. To improve the performance of pedestrian detection using deep learning methods, many approaches have been investigated. [22, 23] proposed a part deformation deep layer to enhance the model with part detection ability. SDN [24] obtained further performance improvement by learning low-level features and high-level semantic parts simultaneously. The work of [25] showed the effectiveness of R-CNN pipeline in pedestrian detection and achieved satisfactory performance on Caltech [26] and KITTI [27] benchmarks. Scale-aware deep detection models [16] was proposed to handle the scale variety problem of pedestrian detection. SSD [28] used features of multi CNN layers to detect objects across multiple scales, which has been proven to be effective and efficient. Thus we follow the work of SSD to design a more simple yet effective multi-resolution feature learning framework. Inspired by [29], we use several max pooling layers to build the multi-resolution feature pyramid, which is efficient and reduces the model size. Moreover, we explore a multi-receptive field module to extract the features with different context size and concatenate them together for final multi-scale representation learning. 2.3 Occlusion handling methods Occlusion problem contributes to a large portion of detection errors, and considerable methods have been proposed to handle this problem. Some approaches [30, 31] focused on learning specific pedestrian occlusion patterns, respectively. [4] designed the DPM, which learned a set of local and root filters for body parts to handle pose variations. The work of [30] proposed to train a small set of classifiers, where each one is responsible for a specific type of occlusion pattern. As pedestrians usually walk or stand together, some works [32] focused on handling the occlusions caused by overlaps between multiple pedestrians. For instance, [33] proposed a complex Bayesian framework to combine different components. Specifically, they first train the part detectors in a supervised manner and then combined the outputs to train a geometric model. Since semantic segmentation can recognise objects in pixel-level, it is also used by several works [34-36] to handle occlusion. [35] proposed a segmentation-aware representation explicitly models and accounts for the supporting pixels for the object within the bounding box, which makes the model more robust to occlusion. Although pixel-level segmentation is effective for occlusion handling, it needs extra segmentation annotation and greater computational costs. Unlike those methods, our approach builds graphs on the high-level CNN feature maps and uses GCN [37] to propagate features across different nodes, which does not need extra supervision and is easy to implement. 3 Methodology The whole proposed framework is shown in Fig. 2. First, the image is input to the backbone CNN network, for example, the Visual Geometry Group (VGG) [38] network for feature learning. Then the multi-resolution and multi-receptive field feature pyramid is built by the proposed MRPP module. The MRPP module contains four max pool layers to downsample the spatial size of the final VGG feature maps and outputs five feature maps with different spatial resolutions. These five feature maps and one VGG features are then inputted to the GCN module for occlusion handling. The GCN processed six feature maps are then used to output the detection results by using the SSD detection algorithm. The final detection results are obtained after the non-maximum suppression module. Fig. 2Open in figure viewerPowerPoint Illustration of the whole proposed detection framework. The final feature pyramid consists of six feature maps with different spatial resolution are shown in bottom right corner of Fig. 2 In the following sections, we will introduce the MRPP module, the GCN [37] occlusion handling module, and the final detection module in detail, respectively. 3.1 Multi-receptive pooling pyramid module The motivation of the MRPP module is to extract multi-resolution feature maps to handle the scale variety problem. The feature maps of deep CNN layers are with higher semantic level but with lower spatial resolution, which is useful to detect large scale objects but not suitable for small scale ones. However, feature maps of relatively shallow CNN layers are with higher spatial resolution, which performs better for small-scale object detection. Different from the previous SSD-based methods, we use the max pooling to construct the multi-resolution feature pyramid, which reduces the number of model parameters and is more time-efficient. Moreover, inspired by the work of [39], multi-receptive field feature are learned with different dilated rate convolution layers. With this module, features of different context information are used to detect pedestrians. Specifically, we formulate this module as follows. Writing the output feature maps of VGG as and , and the output of the Inception [40] module can be represented as (1) where , and denote the convolution in Inception module with kernel size , and . It is worth mentioning that the is implemented with one and two convolution layers for efficiency as shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Illustration of the Inception module used in this work With the Inception module, three feature maps with multiple local contextual information are extract. Inspired by the human visual systems that the size of human population receptive field (pRF) increases with eccentricity, we further input these features to the multiple receptive field feature learning module. The multi-receptive and multi context features Fmrc are computed by (2) where , , and are dilated convolutional layers with dilated rate 1, 3 and 5. Although multi-receptive field and multi-context features are useful for improving the detection performance, these features are still not good at detecting objects with varieties of scales. Following the idea of SSD [28], we aim to build a feature pyramid with multiple spatial scale features. Traditional SSD-based methods use several convolutional layers with stride 2 to reduce the feature spatial size, which needs more computational cost and storage space. Considering this problem, we construct the multi-receptive field and multi-context feature pyramid with the max pooling layers. Specifically, for each of the feature map , and , five max-pooling layers with stride 2 are stacked to generate five different spatial size feature maps. The illustration of this process is presented in Fig. 4. Fig. 4Open in figure viewerPowerPoint Illustration of the MRPP feature learning module At last, a total number of 15 (3 × 5) feature maps are obtained including three feature pyramids . Then we concatenate feature maps with same spatial size but different context and receptive field together to form the final features for detection. For simplicity, we will take one of the final features computation as an example. With the multi-receptive field and multi-context features as input, three down sampled feature maps can be obtained by (3) where is one of the feature in the first branch feature pyramid. By concatenating the three feature maps together, we can get the first final feature for the following occlusion handling module. This can be formulated as (4) and all the final features can be computed in this way. With these five feature maps and one VGG feature map , six feature maps are obtained for the following occlusion handling module to generate the final detection result. 3.2 Graph convolutional neural networks for occlusion handling Occlusion is one of the most challenging problem which limits the performance of pedestrian detection. Although considerable methods have been proposed to handle it, they are still time-consuming and need sophisticated model design. Thus in this work, we introduce a novel graph convolutional network [37]-based occlusion handling method, which is effective and efficient. To clearly describe the proposed method, we first introduce the graph convolutional network. CNNs have been proven to be very effective on representation learning. The features of structured data like the image can be automatically extracted with stacked CNN layers. While for non-Euclidean data, CNN is not suitable for feature learning. GCN is a generalisation of CNN for graph-based data. As graphs can encode complex geometric structures and can be used in many image analysis applications, we choose to build body part graph for occlusion handling in this work. GCN is a first-order approximation of spectral filters on graphs. Spectral-based graph convolution is based on the graph Fourier transform [41]. We begin with the undirected and connected graphs , where V is a set of n nodes, and E is a set of edges. The adjacency matrix of the graph is , and the diagonal degree matrix is denoted as . Then the Laplacian matrix can be computed by , and the normalised definition is (5) where the normalised L is a symmetric positive semi-definite matrix, so its orthonormal decomposition can be formulated as (6) where is the diagonal eigenvalues matrix, and U is the matrix of eigenvectors. Then the spectral graph convolution in Fourier domain can be defined as (7) where is a function of the eigenvalues of the normalised Laplacian matrix L. However, this form of graph convolution need high computational cost, which makes it not scalable for large graph applications. Hammond et al. proposed to approximate the by a truncated order Chebyshev polynomials : (8) where . The Chebyshev polynomials are defined as (9) As , the graph convolution on input signal can be represented as (10) By approximating and , the graph convolution can be formulated as (11) where is a matrix of parameters and X is a C -dimensional feature vector with shape . By stacking multiple GCN layers, representative features can be learned and propagated through the graph. In this work, we build graphs on the final feature maps of the single-shot multi-box detector. For example, a graph is built on every sliding window of as shown in Fig. 5. We assume that three feature vectors in the graph represent ‘head’, ‘body’ and ‘leg’ of a person. From prior knowledge, ‘head’ and ‘leg’ are critical to determine whether a person exists or not. Thus we build the graph to propagate features from ‘head’ to the other two parts and from ‘leg’ to ‘head’. The adjacency matrix of this graph is (12) and the degree matrix of the graph is (13) Fig. 5Open in figure viewerPowerPoint Illustration of the proposed GCN-based occlusion handling module In practice, to implement the graph convolution, we first use a convolution layer on and for feature transformation. Then sliding window method are used to sample feature vectors and build graphs densely. Finally, on each graph the normalised adjacency matrix is multiplied on the convolved feature vectors. This can be formulated as (14) where is the graph (k total graphs) on the feature map X. 3.3 Multi-task training for single-shot detection As the proposed method is based on the SSD detector, we will first introduce the SSD training method. SSD detector uses multiple feature maps for generating different scale detection results. For each position on the final feature maps, a set of different scales and aspect ratios default boxes are used to match the detection result and regress the final box coordinates. Thus during training we need to determine which prior boxes correspond to ground truth and train the model accordingly. The whole proposed framework can be trained in an end-to-end manner. The training objective function consists of two parts. The first one is the classification loss, which is a multiple classes softmax loss and can be formulated as (15) where is an indicator for the sample in class j, and y is the ground truth class label. is the predicted softmax output. (16) For the bounding box regression loss, we use the widely used smooth loss. Specifically, we regress the offsets for the center point and width (w), height (h) between the prior defined bounding boxes and the predicted ones. This loss can be represented by (17) where and are the predicted and ground truth bounding box parameters, respectively. This localisation loss encourages the predicted bounding boxes to match the pre-computed ground truth ones. The whole network is trained with the classification and localisation loss simultaneously, and the total loss function can be formulated as (18) where the parameter is used to balance the two loss terms. 4 Experiments To evaluate the proposed method comprehensively, we have conducted experiments on two public datasets including one general object detection dataset and one large pedestrian detection dataset. On each dataset, we compare the proposed pedestrian detection framework with the recent state-of-the-art methods. In this section, we will first introduce the two public datasets and describe the training details of our network. Then the experimental results are analysed in detail for each dataset. At last, quantitative and qualitative results are presented to visualise the detection performance of this method. 4.1 Datasets and evaluation measurement We use two datasets for the evaluation of the proposed method. One is the PASCAL VOC 2007 dataset, which is a general object detection dataset containing 5011 images for training and validation and 4952 images for test. There are 20 categories in this dataset including the bicycle, bird, person and so on. There are two reasons for us to carry out experiments on this dataset. One is to evaluate the person detection performance. Since we mainly focus on the pedestrian detection task. Another reason is to explore the generalisation performance on other types of objects. Another one is the Caltech pedestrian detection dataset, which consists of ∼250,000 urban scene images. The original dataset contains 11 sets, where set00-set05 are the training sets and set06-set10 are the test sets. To compare with other methods, we use the same settings that only ‘person’ class is used for training. As there are too many images in the original datasets, we sample the training sets with 10 Hz to obtain 42,782 training images. The test set in this work contains 4024 images sampled with 1 Hz from the original test sets. For Pascal VOC dataset, average precision (AP) and mean average precision (mAP) are used for evaluation. AP is defined as the mean precision at a set of 11 equally spaced recall levels [0, 0.1, …, 1], which summarises the property of the precision/recall curve. Following the previous work, we use the standard average-log miss rate (MR) as the evaluation metric for Caltech dataset in this paper. It is the average value of MRs for 9 FPPI (false positives per image) rates evenly spaced in the log-space ranging from 0.01 to 1. According to the occlusion level, the datasets can be divided into none (0% occlusion), partial (0 35% occlusion), and heavy (35 80% occlusion). According to the bounding box height, it can be divided into none [(20, 30) pixels], medium [(30, 80) pixels], and far [(80, 480) pixels]. 4.2 Parameters setup For both datasets, we set the initial learning rate to 4e-3, and stochastic gradient descent optimiser is used for training. We use the same data augmentation strategy with SSD. Four Nvidia TITAN X Pascal cards are used for all the experiments. The batch size for Pascal VOC 2007 dataset is set to 32, and 200 epochs are used to converge. As the object scale of Pascal VOC 2007 is relatively large, we resize all the images to for training and test. However, input size is used for the Caltech dataset, since far pedestrians is too small to detect with input size. Six feature maps are used for ‘SSD 300’, and they are with sizes 38, 19, 10, 5, 3, and 1, respectively. 4.3 Experimental results T