MFP‐Net: Multi‐scale feature pyramid network for crowd counting

IET Image ProcessingEarly View ORIGINAL RESEARCH PAPEROpen Access MFP-Net: Multi-scale feature pyramid network for crowd counting Tao Lei, orcid.org/0000-0002-2104-9298 Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, China The School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, P. R. ChinaSearch for more papers by this authorDong Zhang, Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, China The School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, P. R. ChinaSearch for more papers by this authorRisheng Wang, Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, China The School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, P. R. ChinaSearch for more papers by this authorShuying Li, Corresponding Author angle_lisy@163.com The School of Automation, Xi'an University of Posts and Telecommunications, Xi'an, China Correspondence Shuying Li, The School of Automation, Xi'an University of Posts and Telecommunications, Xi'an 710121, China. Email: angle_lisy@163.comSearch for more papers by this authorWeijiang Zhang, The School of Electronical and Control Engineering, Shaanxi University of Science and Technology, Xi'an, ChinaSearch for more papers by this authorAsoke K. Nandi, orcid.org/0000-0001-6248-2875 The Department of Electronic and Electrical Engineering, Brunel University London, Uxbridge, UK School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, ChinaSearch for more papers by this author Tao Lei, orcid.org/0000-0002-2104-9298 Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, China The School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, P. R. ChinaSearch for more papers by this authorDong Zhang, Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, China The School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, P. R. ChinaSearch for more papers by this authorRisheng Wang, Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, China The School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an, P. R. ChinaSearch for more papers by this authorShuying Li, Corresponding Author angle_lisy@163.com The School of Automation, Xi'an University of Posts and Telecommunications, Xi'an, China Correspondence Shuying Li, The School of Automation, Xi'an University of Posts and Telecommunications, Xi'an 710121, China. Email: angle_lisy@163.comSearch for more papers by this authorWeijiang Zhang, The School of Electronical and Control Engineering, Shaanxi University of Science and Technology, Xi'an, ChinaSearch for more papers by this authorAsoke K. Nandi, orcid.org/0000-0001-6248-2875 The Department of Electronic and Electrical Engineering, Brunel University London, Uxbridge, UK School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, ChinaSearch for more papers by this author First published: 09 June 2021 https://doi.org/10.1049/ipr2.12230AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Although deep learning has been widely used for dense crowd counting, it still faces two challenges. Firstly, the popular network models are sensitive to scale variance of human head, human occlusions, and complex background due to repeated utilization of vanilla convolution kernels. Secondly, the vanilla feature fusion often depends on summation or concatenation, which ignores the correlation of different features leading to information redundancy and low robustness to background noise. To address these issues, a multi-scale feature pyramid network (MFP-Net) for dense crowd counting is proposed in this paper. The proposed MFP-Net makes two contributions. Firstly, the feature pyramid fusion module is designed that adopts rich convolutions with different depths and scales, not only to expand the receptive field, but also to improve the inference speed of models by using parallel group convolution. Secondly, a feature attention-aware module is added in the feature fusion stage. The module can achieve local and global information fusion by capturing the importance of the spatial and channel domains to improve model robustness. The proposed MFP-Net is evaluated on five publicly available datasets, and experiments show that the MFP-Net not only provides better crowd counting results than comparative models, but also requires fewer parameters. 1 INTRODUCTION Dense crowd analysis is one of the most challenging tasks in video surveillance, traffic guidance, public safety prevention and control, and intelligent environment design. The task [1] focuses on crowd counting [2-10, 64], crowd image segmentation [11-15], crowd detection and tracking [16-19], and crowd behaviour recognition and localization [20, 21]. Among them, crowd counting is a basic task in the field of crowd analysis and can be also applied to vehicle detection [22, 23], biometric counting [24-26] etc. Recently, a large number of approaches used for crowd counting have been reported, and these approaches can be roughly grouped into three categories: regression-based approaches, detection-based approaches, and density-map-estimation-based approaches. Traditional approaches for crowd counting mainly depend on regression and detection technique. Detection-based methods are often used to calculate the number of people by detecting the head or appearance of pedestrians by means of dynamic frame detectors [27-31]. These methods are effective in sparse scenes, but they do not perform well in scenes with heavy human occlusions and complex backgrounds. Regression-based methods [32-36] are often used to construct regression models for crowd counting by learning the mapping relationship between shallow features of images and the number of crowds, such as Gaussian-mixture regression, linear regression etc. These regression-based methods can deal with dense crowd counting in complex scenes, but they seriously rely on the feature representation at shallow layers and ignore the spatial features of images, resulting in poor model generalization ability and accuracy. With the rapid development of urbanization, crowd gathering activities are becoming more and more frequent. However, the early crowd counting models focus only on the overall representation of the situation, which is unsuitable for crowd analysis under some complex scenes. For the problem, the density map estimation is better than crowd counting since it estimates the number of people by integrating over the whole image and refines the distribution of local locations. As a result, the crowd counting task has evolved from simple crowd counting to density map estimation that can represent crowd distribution characteristic. Based on this idea, Lempitsky et al. [37] proposed a method based on density map estimation by learning a linear mapping between local features and density maps. However, the method easily suffers from some difficulties because the relationship between features and density maps is usually nonlinear. In order to reduce the difficulty of learning linear mapping relationship, Pham et al. [38] proposed a nonlinear mapping algorithm by utilizing spatial structure information, which applies random forest regression for semi-automatic training of the model. However, the algorithm provides a low accuracy for crowd counting under scenes with high density distribution due to the reliance on manually extracted low-level features. Currently, deep learning is the most popular technique in computer vision due to the achievement of hierarchical feature representation. Based on the deep learning technique, Fu et al. [39] firstly applied general Convolutional Neural Network (CNN) to crowd counting by evaluating crowd density map. However, the general CNN suffers from the problem of loss of image spatial information caused by the employment of fully connected layer for image segmentation. To solve this problem, Fully Convolutional Neural Network (FCN) [40] uses convolution layer instead of fully connected layer to achieve end-to-end pixel-level classification. Since FCN achieves better image segmentation than traditional algorithm [41, 42], it becomes the most popular backbone for image segmentation tasks [43]. Currently, many researchers have applied FCN and improved FCNs to the field of dense crowd analysis, and these FCN-based methods are roughly classified into three categories in terms of model architecture and tasks, that is, multi-branch network for single-task, joined multi-branch networks for multi-task, cascade network for single-task. Multi-branch networks usually use convolutional kernels with different sizes to capture multi-scale image information under different receptive fields. Based on this idea, Zhang et al. [5] designed a multi-column CNN(MCNN) consisting of three encoders, where 3 × 3, 5 × 5 and 7 × 7 convolutional kernels are used for these encoders, respectively. After that, multi-scale features of human head from three encoders are directly concatenated and fused. Finally, the predicted density maps are obtained by a decoder using 1 × 1 convolution. Although MCNN provides better feature representation than general CNNs due to the utilization of multi-scale features, when the networks go deeper, it will require more parameters and higher computational cost. To solve the problem, Zeng et al. [3] used the inception structure to extract multi-scale image features and used 1 × 1 convolution for dimensionality reduction, which to some extent reduces the number of parameters and computational cost of MCNN. Furthermore, Sam et al. [2] presented a switching CNN that includes three independent branching networks, where the input data is fed into one of the branching networks in terms of density rank of input data. More related works can be seen in [22, 44]. Since multi-task learning is helpful for improving the generalization ability of networks on the estimation of crowd density map, many researchers tried to design multi-branch networks for crowd density map estimation. Sindagi et al. [6] proposed a joining learning model used for two tasks, that is, density classification and density map estimation. By using high-level density prior knowledge, the proposed model can accelerate learning and improves prediction accuracy. To further improve prediction accuracy, Gao et al. [45] presented the Perspective Crowd Counting Network (PCC-Net) that consists of three parts: advanced density classification, density map estimation, and image semantic segmentation. For multi-task learning, semantic segmentation effectively distinguishes the background and foreground, and thus improves density map estimation and generalization ability of the network. Based on the idea, some researchers designed multi-task learning frameworks for pixel-level domain adaptation and density map estimation [8, 10, 11, 46]. These frameworks employ generative adversarial networks (GANs) to reduce the difference of scene changes between synthetic and real data, which further reduces the dependence of deep learning on real data. Wang et al. [11] proposed Spatial Fully Convolutional Network (SFCN) that uses dilated convolution and a spatial encoder to incorporate global contextual information leading to higher accuracy for crowd density map estimation. However, multi-task networks depend on richer label data that are uneasily obtained. Cascade networks can provide better feature representation since deeper convolutional layers are often deployed in cascade networks. Li et al. [47] presented a single-column deep network structure (CSRNet) that extracts multi-scale contextual information using dilated convolutional kernels of small-size. The CSRNet greatly improves the density map estimation and counting accuracy for crowd images. Jiang et al. [48] designed hierarchical decoders for different encoding stages and employed dense skip-connections to promote the fusion of multi-scale features, thus improving the quality of predicted density maps. Meanwhile, researchers developed relevant learning strategies based on CNNs to obtain prior knowledge before training, allowing the model to learn incrementally from easy to difficult [9, 49, 50]. In addition, for density map estimation, the removal of background noise is important. Since the attention mechanism allows the model to dynamically focus on the important positions in an image, it can enhance the feature representation of networks. Based on the idea, researchers [51-55] often incorporated an attention module into networks to capture information of interesting regions while ignoring irrelevant information. Although these networks show strong robustness on background noise, they do not consider the relationship between feature channels and feature spatial. Compared with multi-branch networks mentioned above, the cascade networks often provide higher accuracy for the crowd density map estimation. Although both cascade networks and multi-branch networks are successful for crowd counting and density map estimation under some simple scenes, they usually suffer from difficulties caused by complex scenes such as viewing angle of far distance, the scale variance, noise, human occlusions, complex background etc. Figure 1 shows some complex scenes. Despite cascading larger convolutional kernels for multiscale feature extraction can improve the prediction effect, it still causes some new problems, such as more parameters, higher computational cost, and more difficult training. Inspired by [56, 57], we propose a multiscale feature pyramid network (MFP-Net) for crowd counting and density map estimation. The main contributions of this paper are given as follows: (1) For network encoding, we present a feature pyramid fusion module (FPFM). The FPFM employs multiple convolution kernels with different depths and dilation rates to perform group and parallel operations on the input feature maps, which can effectively capture multi-scale contextual information and obtain better feature representations. (2) For feature fusion, we present a feature attention-aware module (FAAM). The FAAM can reduce the effect of background clutter, improves the robustness of the model by dynamically paying attention to interesting regions in images, and leans the visual correlation between spatial and channels. (3) The proposed MFP-Net shows fast training and inference speed due to the utilization of parallel convolution, and provides better crowd counting and density map estimation due to the utilization of adaptive multi-scale feature fusion. FIGURE 1Open in figure viewerPowerPoint The first row is the samples from the NWPU-Crowd dataset [7] and the second row is the ground truth density maps The remainder of the paper is organized as follows. Section 2 describes in detail the structure and advantages of the MFP-Net. Section 3 focuses on the ablation studies and the analysis of comparative experimental results. In Section 4, summary and discussion are presented. 2 METHOD In this paper, we propose a multi-scale feature pyramid network (MFP-Net) and apply it to the field of crowd counting and density map estimation. Figure 2 shows the architecture of MFP-Net that consists of feature extraction layer, feature pyramid fusion layer and feature attention-aware layer. Firstly, considering that VGG16 has a low training cost and high performance, we choose the first 10 layers of VGG16 as the feature extraction layer of MFP-Net. Secondly, the feature maps obtained by feature extraction layer are fed into the feature pyramid fusion module that adopts pyramid group convolution to efficiently perform multi-scale feature extraction and obtain different levels of fine-grained information from input images. Then, the obtained multi-scale feature maps are fed into the feature attention-aware module that can capture more meaningful features and achieve adaptive fusion of local and global information, which is helpful for improving feature representation and the robustness of the network on the tasks of crowd counting and density map estimation. Finally, the feature maps are restored to the original image size by using bilinear interpolation without parameters. In fact, both the feature pyramid fusion module and the feature attention-aware module are used for feature fusion. However, the former aims to extract multi-scale information from crowed images, but the latter aims to capture the key information and suppress the influence of background noise. FIGURE 2Open in figure viewerPowerPoint The architecture of the proposed MFP-Net. First, we use VGG16 as the backbone. Secondly, the feature pyramid fusion module is used to extract multi-scale information and then use the feature attention-aware module for feature fusion 2.1 Feature pyramid fusion module Dense crowd images often have a complex background and a large variation in the scale of crowd object proximity and distance. Thus, vanilla convolution has two drawbacks. One is vanilla convolution has a fixed receptive field and cannot efficiently extract multi-scale information in crowd images. The other is the stacking of different scale vanilla convolutional kernels causes the increase of the number of network parameters and the reduction of network robustness. To solve these two problems, we design the feature pyramid fusion module to capture the multi-scale information in crowd images. Roughly, we first split the feature maps into multiple blocks, and then perform 3 × 3 group convolution with different dilation rates on each block as shown in Figure 3. The feature pyramid fusion module performs different levels of filtering operations on the input feature maps by using group convolution, which captures different multi-scale contextual information in parallel computation. Moreover, the group convolution can reduce the computational cost and thus improves the inference speed of networks. FIGURE 3Open in figure viewerPowerPoint The architecture of the feature pyramid fusion module In Figure 3, the feature pyramid fusion module includes four pyramidal convolution layers. In each layer, the feature maps are first divided into proportional blocks. And then each block is performed group dilated-convolution. Specifically, the channel number of the input feature map is M. In the first layer, we divide feature maps into 4 blocks, and the channel number of each block is C 1 , C 2 , C 3 , C 4 , respectively, where C 1 + C 2 + C 3 + C 4 = M . All convolutional kernels are 3 × 3 with different dilation rates r ( r =1,2,3,4), where the number of groups G increases by 2 n in pyramid-shape, for example, G = ( 2 0 , 2 1 , 2 2 , 2 3 ) . For the second layer, we divide the feature maps into three blocks. Similarly, the size of all convolutional kernel is 3 × 3, and the dilation rate r and the number of groups G are incremented sequentially starting from 1. The third layer is divided into two blocks. For the last layer, we adopt vanilla convolution with G = 1 . Here, we define the input feature maps as x and the output feature maps as y, then y i ( x ) = P G c o n v ( x , N i , G i , r i ) , i = 1 P G c o n v ( y 1 ( x ) , N i , G i , r i ) , i = 2 ⋮ P G c o n v ( y L − 1 ( x ) , N i , G i , r i ) , i = L , (1)where P G c o n v ( x , N i , G i , r i ) is the pyramid group dilated-convolution, L is the number of layers, N i is the number of blocks, r i is the dilation rate, and G i is the number of groups in each convolution operation. Note that L , N i , r i , and G i are hyperparameters that can be adjusted according to different tasks. As shown in Figure 3, our presented feature fusion module is pyramid-shaped, and the number of groups as well as the dilation rate increases in a pyramidal pattern. In such a way that different levels of fine-grained features can be extracted, and multi-scale features can be clustered at the top of the pyramid. Compared with the vanilla convolution, the proposed pyramid group dilated-convolution can extract richer multi-scale image features due to the utilization of dilated convolution with different dilation rate. Compared with multi-scale convolution, the proposed method requires fewer parameters and lower computational cost due to the utilization of pyramid grouping. Therefore, the feature pyramid fusion module both extracts multi-scale information and simultaneously improves the inference speed of the network. Figure 4 shows the comparison of vanilla convolution and group convolution, where the number of groups is 2. Obviously, the computational cost is greatly reduced when we use group convolution instead of vanilla convolution. The computational cost of group convolution is denoted by F, and F G , K , C i n , C o u t = K 2 × C i n × C o u t × H × W G , (2)where G is the number of groups, K is the size of the convolution kernel, C i n and C o u t are the number of input and output feature maps, respectively, H and W are the height and width of the feature maps. In fact, when performing group convolution, a larger value of G corresponds to a lower computational cost F ( G , K , C i n , C o u t ) . For the feature pyramid fusion module, the number of groups, blocks and dilation rates gradually decrease in a pyramidal way as the network depth increases, which aims to achieve a balance between feature representation and inference speed. Although group convolution is able to improve inference speed of networks, it leads to feature loss since feature maps are grouped for convolution. Consequently, the pyramid group dilated-convolution requires a low computational cost: F ( N , G , K , C i n , C o u t ) = ∑ i = 1 N K i × C i n i × C o u t i × H × W G i , (3)where N is the number of proportionally divided blocks. C i n i and C o u t i are the number of input and output feature maps in the ith group, respectively, and C i n i ∈ C i n , C o u t i ∈ C o u t . G i is the number of groups in the i-th block, G i ∈ G . Obviously, it can be concluded from (3) that the dilated convolution expands the receptive field without losing the resolution of the feature maps. Meanwhile the group parallel convolution can reduce computational cost. FIGURE 4Open in figure viewerPowerPoint Comparison of vanilla convolution and group convolution In conclusion, our presented feature pyramid fusion module not only provides better feature representation caused by multi-scale feature extraction, but also achieves fast inference speed and requires lower computational cost due to the employment of pyramid group convolution. 2.2 Feature attention-aware module For the task of crowd density estimation, congested scenes have disturbing factors such as scale, perspective, occlusion, brightness etc. Previous works extract features as equally important and fuse them indiscriminately, resulting in poor robustness. However, the cluttered background is often identified as crowd during the actual density map estimation that affects the accuracy of the model prediction. Therefore, it is a challenge to effectively identify confusable features. The attention mechanism can enhance the robustness of feature representation by focusing on important regions and key channels of feature maps. Liu et al. [51] adopted attention mechanism to obtain local location information in feature maps yet ignores the global correlation between feature channels. Sindagi et al. [58] injected foreground and background segmentation information into the counting network through an attention mechanism but did not consider the relationship between each location in the image. For these problems, we present a feature attention-aware module that integrates both position and channel attention mechanism. The position attention-aware focuses on the spatial correlation between each position in the image to capture the global correlation of images, which is helpful for mapping the distribution pattern of crowd density. The channel attention-aware focuses on the correlation between different channels in feature maps and emphasizes the interdependence between feature maps by assigning different weights. Due to the semantic relevance of spatial and channels, we use both channel and position attention and then adaptively fuse the obtained feature maps. Figure 5 illustrates the structure of feature attention-aware module. The structure that contains two types of attention modules that explore local and global contextual information by constructing associations between features to improve feature representation for crowd counting and density estimation. On the one hand, the position attention-aware module encodes a wider range of contextual information than convolution operation, thus enhancing the representation of local features. On the other hand, the channel attention-aware module reduces the effect of useless feature maps caused by background noise. The feature map G ∈ R C × H × W output from the feature pyramid fusion module is fed into a convolution layer to obtain two feature maps ( G 1 , G 2 ) that are reshaped to R C × N , where C is the number of channels of feature maps, H × W denotes the spatial dimension, N = H × W denotes the number of image pixel points. We then perform a matrix multiplication between the transpose of G 2 and G 1 , that is, ( G 2 C × N ) T × G 1 C × N . Finally, the obtained results are fed into a softmax layer to obtain the spatial correlation matrix W ( s ) ∈ R N × N : w s i j = e x p ( G 2 j · G 1 i ) ∑ i N e x p ( G 2 j · G 1 i ) , (4)where w s i j denotes the correlation measure between the jth position and the ith position in a feature map. we perform a matrix multiplication between G 1 and W ( s ) , that is, G 1 × W ( s ) and then reshape the result to R C × H × W to obtain the position attention-aware maps G ( s ) . Next, we multiply the obtained result by a spatial scale parameter μ and perform an element-wise summation with G to obtain the result G s ′ ∈ R C × H × W : G s ′ = μ ( G 1 C × N × ( G 2 C × N T × G 1 C × N ) ) + G C × H × W , (5)where the spatial scale parameter μ is a parameter learned gradually from 0. FIGURE 5Open in figure viewerPowerPoint The structure of feature attention-aware module In addition, each channel mapping of high-level features can be viewed as a class-specific response with different semantic interrelated. By exploiting the interdependencies between channel mappings, we can emphasize the interdependent feature mappings and improve the semantic-specific feature representation. Therefore, we present a channel attention-aware module to explicitly model the interdependencies between channels. We first feed F ∈ R C × H × W that is the output of the feature pyramid fusion module into a convolution layer to get two feature maps F 1 and F 2 , { F 1 , F 2 } ∈ R C × H × W . Secondly, we reshape them to R C × N , where N = H × W denotes the number of pixels in an image. Further, we perform the multiplication of the matrix F 2 C × N × ( F 1 C × N ) T . The global correlation matrix W ( c ) ∈ R C × C is then calculated via a softmax slayer, and it is defined as: w c i j = e x p ( F 2 j · F 1 i ) ∑ i C e x p ( F 2 j · F 1 i ) , (6)where w c i j denotes the value of the weight of the jth channel on the ith channel. Then we perform the matrix multiplication W ( c ) × F 2 and reshape it to R C × H × W . As a result, the channel attention feature map F ( c ) is obtained. After that, we multiply the obtained result by a channel scale parameter ρ and perform an element-wise summation with F to obtain the result F c ′ ∈ R C × H × W : F c ′ = ρ ( ( F 2 C × N × F 1 C × N T ) × F 2 C × N ) + F C × H × W , (7)where ρ is the channel scale parameter learned gradually from 0. In (7), we can see that a feature map is obtained from the summation of the original features and the attention features. Such an approach preserves more fine-grained information and helps models to enhance the feature representation between channels. The final feature maps via the feature attention-aware module are defined as: B 2 C × H × W = F c ′ ⊕ G s ′ , (8)where F c ′ denotes the finally obtained channel correlation feature maps and G s ′ denotes the spatial correlation featur