Crowd counting with segmentation attention convolutional neural network

IET Image ProcessingVolume 15, Issue 6 p. 1221-1231 ORIGINAL RESEARCH PAPEROpen Access Crowd counting with segmentation attention convolutional neural network Jiwei Chen, Jiwei Chen Institute of Intelligent Machines,Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, People's Republic of China Department of Automation, University of Science and Technology of China, Hefei, Anhui, People's Republic of ChinaSearch for more papers by this authorZengfu Wang, Corresponding Author Zengfu Wang zfwang@ustc.edu.cn Institute of Intelligent Machines,Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, People's Republic of China Department of Automation, University of Science and Technology of China, Hefei, Anhui, People's Republic of China Correspondence Zengfu Wang, Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, People's Republic of China. Email: zfwang@ustc.edu.cnSearch for more papers by this author Jiwei Chen, Jiwei Chen Institute of Intelligent Machines,Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, People's Republic of China Department of Automation, University of Science and Technology of China, Hefei, Anhui, People's Republic of ChinaSearch for more papers by this authorZengfu Wang, Corresponding Author Zengfu Wang zfwang@ustc.edu.cn Institute of Intelligent Machines,Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, People's Republic of China Department of Automation, University of Science and Technology of China, Hefei, Anhui, People's Republic of China Correspondence Zengfu Wang, Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui, People's Republic of China. Email: zfwang@ustc.edu.cnSearch for more papers by this author First published: 12 January 2021 https://doi.org/10.1049/ipr2.12099Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Deep learning occupies an undisputed dominance in crowd counting. This paper proposes a novel convolutional neural network architecture called SegCrowdNet. Despite the complex background in crowd scenes, the proposed SegCrowdNet still adaptively highlights the human head region and suppresses the non-head region by segmentation. With the guidance of an attention mechanism, the proposed SegCrowdNet pays more attention to the human head region and automatically encodes the highly refined density map. The crowd count can be obtained by integrating the density map. To adapt the variation of crowd counts, SegCrowdNet intelligently classifies the crowd count of each image into several groups. In addition, the multi-scale features are learned and extracted in the proposed SegCrowdNet to overcome the scale variations of the crowd. To verify the effectiveness of this proposed method, extensive experiments are conducted on four challenging datasets. The results demonstrate that the proposed SegCrowdNet achieves excellent performance compared with the state-of-the-art methods. 1 INTRODUCTION Recently, crowd counting has become a research hotspot owing to its wide applications including public safety management [1], crowd analysis [2] and urban planning [3]. Besides, the methods of crowd counting have important references in vehicle counting, cell counting, and other objects counting [4]. However, due to the crowd's complexity such as severe occlusion, high diversity, scale variation, view-point variation, and non-uniform distribution, the accuracy of crowd counting still has significant room for improvement. Lots of efforts have been done in crowd counting. We generally classify these methods into detection-based methods and regression-based methods. The detection-based methods [5-8] obtain the crowd count by counting the number of positive detections. The regression-based methods [6, 9-11] utilise the mapping between extracted features and cropped patches to regress the crowd count. However, the aforementioned methods are complex, because their features need to be extracted by extremely numerous and cumbersome designs. In recent years, deep learning has obtained breakthroughs in many fields [12-14]. Many CNN-based methods [15-17] have been proposed in crowd counting. They learn the non-linear mapping between training images and crowd counts. Researchers employ these CNN models to generate a density map that records the count and spatial information of the crowd at each pixel location, then the density map is integrated to get the crowd count. Although deep learning has achieved great improvement in crowd counting, we find that the background is extremely complex in the crowd scene, and we believe that the accuracy of crowd counting can be further improved with the segmentation attention mechanism which highlights the foreground, suppresses the background and makes the proposed SegCrowdNet pay more attention to the foreground. Some example images are shown in Figure 1. Let's carefully distinguish the foreground and background in crowd counting. We obtain the crowd counting result by counting the number of the human head. The human head belongs to the foreground. Everything apart from the human head belongs to the background. Therefore, the background is extremely complicated. In Figure 1, we can also observe that the crowd count changes dramatically across images, which is the second challenging problem. To address these two challenging problems, firstly, we predict a segmentation map to adaptively highlight the human head region and suppress the non-head region. With the guidance of the segmentation results, the adaptive attention weights are used in the estimation of the density map to guide our network called SegCrowdNet to pay more attention to the human head region and generate a highly refined density map. Integrating the refined density map can get the crowd counting result. Unfortunately, the present datasets don't provide the ground truth of segmentation. We propose a simple but effective method in which the ones template is pasted on a binary map to encode the ground truth of segmentation. Secondly, to adapt the large variation of crowd counts, we utilised a classification task where the crowd count of each image is classified automatically. Meanwhile, to extract the multi-scale features to adapt the multi-scale crowd, our proposed SegCrowdNet not only utilises different convolution kernels to encode the image but also fuses rich hierarchies from different depths of convolutional layers where the lower layer can extract discriminative features of the pedestrian and the higher layer can learn semantic concepts of the same pedestrian. FIGURE 1Open in figure viewerPowerPoint Two challenges for crowd counting. (1) Only the human head (highlight area) belongs to the foreground. The background is extremely complex. (2) The crowd count in each image varies greatly. GT represents the crowd count of the image This paper proposes a novel end-to-end framework called SegCrowdNet. To the best of our knowledge, the proposed SegCrowdNet is the first network to utilise the segmentation attention mechanism in crowd counting. It adaptively highlights the human head region and suppresses the non-head region by way of optimising a novel loss. With the guidance of the segmentation attention mechanism, the proposed SegCrowdNet pays more attention to the human head region and automatically encodes the highly refined density map. Our proposed SegCrowdNet can also automatically adapt the variation of crowd counts by learning a classification function. Extensive experiments are conducted on ShanghaiTech Part_A dataset [15], ShanghaiTech Part_B dataset [15], UCF_CC_50 dataset [11] and WorldExpo'10 dataset [18]. The results demonstrate that our proposed method outperforms many state-of-the-art methods. 2 RELATED WORK The researches [19-25] about crowd counting are too rich to elaborate all of them. Next, we briefly review some of them. 2.1 Counting by detection In early researches, many kinds of detection frameworks are proposed in crowd counting. The pivotal features from human body are extracted by the well-trained classifiers such as HOG [5], Random Forest [6], and Haar wavelets [7], then the classifier outputs the positive samples. The total number of positive samples represents the crowd count. [26-28] complete the crowd counting task by using the aforementioned method, and [22] leverages adaptive thresholds to binarize the image to detect the crowd. They can get a good result in the sparse crowd scene. However, when the crowd density becomes high, some persons are too small to be detected. 2.2 Counting by regression In the highly congested crowd scene, the regression-based methods are usually chosen. They have two important stages. The low-level features are first extracted from the input image, then the crowd count is regressed according to the low-level features. To record the space information of the crowd, Lempitsky et al. [10] proposed a good method that could learn a linear mapping between local area features and corresponding density maps. Based on Lempitsky's work, Pham et al. [6] proposed a more applicable method. The linear mapping was replaced with a non-linear mapping based on random forest regression. In [24], SVM is proposed to map the features extracted by AdaBoost [29] to the crowd counting result. 2.3 Counting by CNN In recent years, CNN has achieved great success in the computer vision task including crowd counting. Boominathan et al. [30] combined the deep and shallow CNN to extract large scale features to overcome the scale variation. Zhang et al. [15] proposed a three-branch convolutional neural network to encode the density map with different scales. Based on Zhang's work, Sam et al. [16] designed a classifier to select the most appropriate branch to generate the density map according to the density variation in each image. Similarly, Hossain et al. [31] proposed a scale-aware method to make each branch in their model focus on a particular scale. Mahmood et al. [23] designed an Encoder–Decoder CNN to learn the head heatmap where human heads can be segmented. Shen et al. [32] introduced the scale-consistency model to alleviate the blurry effects in the estimated density map. Sam et al. [33] designed the TDF-CNN which contains the bottom-up and top-down CNNs to obtain initial accurate predictions. Marsden et al. [4] proposed a CNN model that can count the crowd, wildlife, vehicles, and cells. Wang et al. [34] designed a collector and labeler to generate the crowd data with corresponding labels to reduce the overfitting caused by limited training crowd data. Xu et al. [21] proposed a segmentation branch based on the probability to guide their framework to encode the final density map Chen et al. [25] proposed a crowd attention network to avoid the enormous misjudgements caused by the scene complexity. This paper proposes a novel segmentation attention mechanism to guide the end-to-end architecture called SegCrowdNet to pay more attention to the human head region. Our segmentation is based on the dice coefficient and we employ the CNN and ground-truth density map to fine tune the final density map. Moreover, our proposed SegCrowdNet can adapt the variation of crowd counts by a classification task. 3 METHODOLOGY 3.1 SegCrowdNet architecture An overview of the proposed SegCrowdNet can be seen in Figure 2. The backbone is designed to extract multi-scale features. The proposed SegCrowdNet contains the classification task, the segmentation task and the density estimation task. FIGURE 2Open in figure viewerPowerPoint The proposed architecture of our SegCrowdNet. It is a multi-task model including the classification task (Cla-task), segmentation task (Seg-task), and density estimation task (Des-task). The convolutional layers' parameters are represented as "Con(kernel size)-(number of filters)-(dilation rate)" ⨁ represents the element-wise add. The estimated density map (outlined in blue) is encoded in the intermediate supervision process The configurations of the backbone are shown in Figure 3. To extract multi-scale features to overcome the scale variations of the crowd, firstly, four different receptive fields are inserted into our system at the beginning of this backbone. Each of them has 16 filters. The input image is mapped at different scales by them synchronously, then the results are fed to the following modules. Secondly, inside this module, we design several 2x2 max-pooling layers to extract multi-scale features. Thirdly, we fuse the feature maps with complementary information from different depths of convolutional layers. Every convolutional layer is followed by ReLU. In order to increase the depth of the network to enhance its learning ability without introducing too many parameters, the parameters of the shared module are shared to alleviate overfitting caused by excessive parameters. Since the dilated convolution [35] can increase the receptive field with fewer parameters, the dilated convolution is widely used in our SegCrowdNet. FIGURE 3Open in figure viewerPowerPoint The proposed architecture of the backbone. The convolutional layers' parameters are represented as "Conv(kernel size)-(number of filters)- (dilation rate)". The kernel size of the max pool is 2 × 2 with stride 2 In Table 1, it can be observed that the crowd count changes greatly across images in the "Range" column. For example, the UCF_CC_50 dataset only contains 50 images. However, the crowd counts range from 94 to 4,543. So the classification task is employed. The classifier can automatically learn the crowd count distribution to adapt the variation of crowd counts. Inspired by [36], the crowd count of each image in the crowd dataset is quantised into several groups. As shown in Fig 2, the fully connected (FC) layers are connected at the end of the backbone. The two fully connected layers followed by PReLU separately have 64 neurons and 5 neurons. The 5 neurons represent five count groups. To avoid the distortion of images and maintain the original distribution of the crowd, we do not resize the input image. The Spatial Pyramid Pooling (SPP) [37] layers are placed between convolutional layers and FC layers. Arbitrarily sized feature map extracted from the input image can be fed to the SPP layer, and the SPP layer produces fixed size outputs to feed the FC layers. In the classification task, the crowd counts of each dataset are classified into five count groups to adapt the variation of crowd counts. We choose to minimise the cross-entropy loss to optimise this process. TABLE 1. Statistics of the training datasets: Num represents the number of images; Range represents the range of crowd count in each image; Average represents the average crowd count; Total represents the total number of labelled people Dataset Resolution Color Num Range Average Total ShanghaiTech Part_A different RGB,Grey 482 [33, 3139] 501.4 241,677 ShanghaiTech Part_B 768 × 1024 RGB 716 [9, 578] 123.6 88,488 UCF_CC_50 different Grey 50 [94, 4543] 1279.5 63,974 WorldExpo'10 576 × 720 RGB 3980 [1, 253] 50.2 199,923 Two other tasks are also shown in Figure 2. In the segmentation task, the segmentation map which adaptively emphasises the head region and suppresses the non-head region is encoded with the supervision of ground-truth segmentation. In the density estimation task, the estimated density map (outlined in blue) and the final estimated density map are predicted with the supervision of ground-truth density map. Integrating the final estimated density map can get the crowd count. As shown in Figure 2, we add the segmentation map to the estimated density map (outlined in blue) and the results are fed to the following convolution layers and ReLU layers to automatically encode the final estimated density map. With the guidance of the segmentation map where the human head regions have higher weights, more attention is paid to the human head region in the density estimation task. In the segmentation task, a novel loss based on the dice coefficient is employed. The novel loss and the ground truth of segmentation will be elaborated in Sec 3.2. In the density estimation task, the Euclidean distance loss is employed to optimise the estimated density map and the final estimated density map. 3.2 Model learning 3.2.1 Ground truth generation The ground-truth density map is extremely important for the supervised methods. For any training image, the 2D point p located at the centre of each human head is provided. The ground-truth density map is encoded by employing a normalised Gaussian kernel centred on each point p, which is defined as: D ( c ) = ∑ p ∈ A j N ( p ; μ , σ ) , (1)where c represents the pixel location. A j represents a series of 2D points annotated in the image j. N ( p ; μ , σ ) represents the normalised Gaussian kernel with mean μ = 0 and isotropic variance σ = 4 . The window size of the Gaussian kernel is 15× 15. We utilise the above simple method to generate the ground-truth density map to ensure that the improvement of the result comes from our novel method instead of the innovation of generating a ground-truth density map. It is extremely expensive to manually label the ground truth of segmentation, owing to the huge amount of people in datasets. For example, there are 330,165 people in the ShanghaiTech dataset. So a simple but effective method is proposed to encode the ground-truth segmentation map which should have the same foreground and background with the ground-truth density map. First, we construct the ones template that keeps the same size with the Gaussian kernel, which is set to 15× 15. Second, the ground truth of segmentation is encoded by pasting the ones template that is centred on each annotated point p on a binary map. They are visualised in Figure 4. The experiments using different scale templates are conducted on the challenging ShanghaiTech Part_A. Results are given in Table 2. It can be found that performance with 15× 15 is the best. We use the same scale template (15× 15) on all four datasets to show the robustness of our proposed method. TABLE 2. Estimation errors of the template with different sizes on ShanghaiTech Part_A Template size MAE MSE 5 × 5 110.1 160.7 10 × 10 101.3 139.0 15 × 15 68.3 104.1 20 × 20 74.8 114.5 25 × 25 68.5 106.7 FIGURE 4Open in figure viewerPowerPoint Some visualised results of ground-truth density map and ground-truth segmentation 3.2.2 Model optimisation The proposed SegCrowdNet is a multi-task model including classification task, segmentation task, and density estimation task. The multi-task learning [38] can serve as a regularisation to alleviate overfitting, which requires our proposed SegCrowdNet to consider every task synchronously, not only one of them. The proposed SegCrowdNet is optimised by minimising four loss functions in a synergistic manner, including an intermediate supervision loss. In the density estimation task, the Euclidean loss is utilised to optimise the estimated density map (outlined in blue) and the final estimated density map, L i n t = 1 2 U ∑ i = 1 U d ̂ i − D i 2 2 , (2) L d e n = 1 2 U ∑ i = 1 U D ̂ i − D i 2 2 , (3)where d ̂ i represents the estimated density in the intermediate supervision process. D ̂ i represents the final estimated density. D i represents the ground-truth density. U represents the number of pixels in the ground-truth density map. In the segmentation task, inspired by [39, 40], we introduce a novel loss in crowd counting which is based on the dice coefficient. The loss is optimised to predict the segmentation map for the human head. The dice coefficient D ( s i ̂ , s i ) is between 0 and 1. In the process of optimisation, D ( s i ̂ , s i ) is expected to maximise, and the loss L s e g is expected to minimise. They are formulated as: D ( s i ̂ , s i ) = 2 ∑ i U s i ̂ s i ∑ i U s i ̂ 2 + ∑ i U s i 2 , (4) L s e g = 1 − D ( s i ̂ , s i ) , (5)where U represents the total number of pixels in the ground truth segmentation map. s i represents the ith value in the ground truth segmentation map. s i ̂ represents the ith value in the predicted segmentation map. In the classification task, the crowd counts are quantised into five groups. For example, if the crowd counts of a dataset range from 1 to 500 and they are quantised into five classes, the images with populations between 1 and 100 belong to the first class, and the images with populations between 401 and 500 belong to the fifth class. The cross-entropy loss function is utilised, L c l a = − 1 M ∑ a = 1 M ∑ b = 1 K y b a l o g ( y ̂ a ) , (6)where M represents the total number of training samples. K represents the total number of classes. y b a represents the ground-truth class. y ̂ a represents the output of classification. The final weighted loss is given by Equation (7). Empirically, we set λ1 as 0.01. L f i n = L d e n + L i n t + L s e g + λ 1 L c l a . (7) 4 EXPERIMENTS 4.1 Experiment setting We conduct extensive experiments on four challenging datasets. The statistics of the four challenging datasets are summarised in Table 1. Some example frames from these four datasets are shown in Figure 6. In the process of creating training data, nine patches are cropped from each original image randomly, and the size of each patch is 1/4 of the original image. Horizontal flipping and noise addition are also used in the training data. We utilise Adam optimisation with a fixed constant momentum of 0.9 to train our model, and we set the learning rate as 1e-6. FIGURE 5Open in figure viewerPowerPoint The crowd count comparisons of different configurations about the Seg-task on ShanghaiTech Part_A. X-axis: the ID of images, Y-axis: the crowd counts of images FIGURE 6Open in figure viewerPowerPoint Some example frames from training datasets 4.2 Evaluation metric The mean absolute error (MAE) and the mean squared error (MSE) are widely used to measure the crowd counting error on different methods [15, 16, 33, 34]. They are defined as follows: M A E = 1 V ∑ i = 1 V z i − z i ̂ , (8) M S E = 1 V ∑ i = 1 V z i − z i ̂ 2 , (9)where z i represents the ground-truth crowd count. z i ̂ represents the estimated crowd count. V represents the number of test images, 4.3 Ablation study using ShanghaiTech part_A To verify the effectiveness of each task in the proposed SegCrowdNet, we perform ablation studies on ShanghaiTech Part_A [15] which is a large scale and high-density dataset, the dataset contains 482 images with 241,667 annotated persons. In other datasets, similar results can be observed. To investigate the ability of this classification task where the crowd count of each image is classified, we conduct a set of comparative experiments. In one experiment, the classification task can function normally. In the second experiment, the classification task is removed. The estimation errors are given in Table 3. We can observe that the performance with Cla-task is better, with MAE/MSE 12.4/19.9 lower than that without Cla-task. We think that the proposed SegCrowdNet can learn the crowd count distribution to adapt the variation of crowd counts, which contributes to reducing the crowd counting errors. In the proposed SegCrowdNet, we classify the crowd counts of images in each dataset into five categories according to extensive experiments. From Table 4, we find that our proposed SegCrowdNet with five categories perform best. We think that the crowd counting data are limited and five categories are most suitable. Last but not least, we can further observe that adopting the classification task with different categories in Table 4, the MAE is further alleviated compared with the performance without Cla-task in Table 3, which further indicates that the Cla-task can assist in reducing the estimation error of crowd counting. TABLE 3. Estimation errors of different configurations about the Cla-task on ShanghaiTech Part_A Method MAE MSE Without Cla-task 80.7 124.0 With Cla-task 68.3 104.1 TABLE 4. Estimation errors of different categories about the crowd counts on ShanghaiTech Part_A Categories 3 5 7 10 15 MAE 71.3 68.3 77.8 78.4 70.4 The proposed segmentation attention mechanism is the most important idea in this paper. It is included in the segmentation task. To demonstrate the effectiveness of the segmentation task, we also conduct a set of comparative experiments. In one experiment, the segmentation task can function normally. In the second experiment, the segmentation task is removed from our proposed SegCrowdNet and the segmentation attention mechanism is subsequently removed. The qualitative results are illustrated in Table 5. We can observe that the SegCrowdNet greatly reduces the error of crowd counting with the guidance of segmentation results, with MAE/MSE 17.5/24.5 lower than that without Seg-task, which reveals the power of the segmentation attention mechanism. In Figure 5, we demonstrate the estimated crowd count of each image in the comparative experiment. It can be observed that the blue circle (predicted crowd counts with the guidance of the Seg-task) is closer to the red triangle (actual crowd counts) than the green pentagram (predicted crowd counts without Seg-task) in most images, which indicates that the segmentation attention mechanism plays a vital role in alleviating the errors of crowd counting. TABLE 5. Estimation errors of different configurations about the Seg-task on ShanghaiTech Part_A Method MAE MSE Without Seg-task 85.8 128.6 With Seg-task 68.3 104.1 We visualise the effectiveness of the Seg-task in Figure 7. In the third column, it can be observed that the human head regions are highlighted and the non-head regions are suppressed excellently in the estimated segmentation map, which is important to guide our proposed SegCrowdNet to focus on the human head region. In the fourth and fifth columns, we can observe that with the guidance of the segmentation results, the proposed SegCrowdNet pays more attention to the human head region and the crowd counts are predicted beautifully. Similar results can be found in Figure 8. From these results, we can see that every task in the proposed SegCrowdNet is necessary. They work collaboratively to better accomplish the task of crowd counting. FIGURE 7Open in figure viewerPowerPoint Several visualisation results of the ShanghaiTech Part_A. Top Column: Some input images from the ShanghaiTech Part_A dataset. Second Column: The ground-truth segmentation. Third Column: The predicted segmentation map. Fourth Column: The ground-truth density map. Fifth Column: The final estimated density map. GT: The crowd count of the ground-truth density map. Est: The crowd count of the estimated density map FIGURE 8Open in figure viewerPowerPoint Several visualisation results of the ShanghaiTech Part_B. Top Column: Some input images from the ShanghaiTech Part_B dataset. Second Column: The ground-truth segmentation. Third Column: The predicted segmentation map. Fourth Column: The ground-truth density map. Fifth Column: The final estimated density map. GT: The crowd count of the ground-truth density map. Est: The crowd count of the estimated density map 4.4 ShanghaiTech dataset The ShanghaiTech dataset [15] is highly challenging because of its large-scale crowds. It contains 1,198 annotated images with a total number of 330,165 persons. So far the dataset contains the richest annotated crowd information and it is composed of two parts: Part_A with 300 training images and 182 test images, Part_B with 400 training images and 316 test images. Part_A consists of the high-density crowd images downloaded from the internet. Part_B consists of the relatively low-density crowd images collected from Shanghai streets. As shown in Table 6, we compare our proposed method with the other fourteen state-of-the-art methods. In the LBP+RR method, the LBP features were extracted by hand to regress the crowd count. Owing to the scale variation in the crowd, the work of MCNN [15], CAFN [41], TDF-CNN [33], Switch-CNN [16], and CP-CNN [17] employed multi-column CNNs to extract the multi-scale features to cope with the scale variation in the crowd. Instead of using the multi-column CNN, in [31], Hossain et al. proposed the scale-aware attention network to encode the multi-scale density maps. In [42], IG-CNN adapted the multi-scale crowd scenes by increasing its capacity. The work of Cascaded-MTL [36], Marsden et al. [4], DecideNet+R3 [43], and DDCN [44] introduced the multi-task learning to assist in reducing the count estimation errors. The work of SE Cycle GAN [34] designed the data collector and labeler to generate the crowd data with the corresponding labels to alleviate the overfitting caused by limited data for