Foreground segmentation using multiscale convolutional neural network

Electronics LettersVolume 56, Issue 12 p. 597-599 Image and vision processing and display technologyFree Access Foreground segmentation using multiscale convolutional neural network Hui-Ni Fu, Corresponding Author Hui-Ni Fu fuhuini15@nudt.edu.cn School of Computer Science and Technology, National University of Defense Technology, Changsha, 410000 People's Republic of ChinaSearch for more papers by this authorBen-Zhang Wang, Ben-Zhang Wang School of Computer Science and Technology, National University of Defense Technology, Changsha, 410000 People's Republic of ChinaSearch for more papers by this authorHeng-Zhu Liu, Heng-Zhu Liu School of Computer Science and Technology, National University of Defense Technology, Changsha, 410000 People's Republic of ChinaSearch for more papers by this author Hui-Ni Fu, Corresponding Author Hui-Ni Fu fuhuini15@nudt.edu.cn School of Computer Science and Technology, National University of Defense Technology, Changsha, 410000 People's Republic of ChinaSearch for more papers by this authorBen-Zhang Wang, Ben-Zhang Wang School of Computer Science and Technology, National University of Defense Technology, Changsha, 410000 People's Republic of ChinaSearch for more papers by this authorHeng-Zhu Liu, Heng-Zhu Liu School of Computer Science and Technology, National University of Defense Technology, Changsha, 410000 People's Republic of ChinaSearch for more papers by this author First published: 01 June 2020 https://doi.org/10.1049/el.2019.3668Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Foreground segmentation is to segment moving objects from a video or camera. Many approaches have been proposed to perform background subtraction. However, these algorithms could not handle complex scenes like illumination changes, dynamic background and camouflage etc. In this Letter, the authors propose a robust end-to-end convolutional neural network with multiscale features to address these problems. They use only a few training samples with RGB channels and output the corresponding foreground mask, which could save millions in labelling ground-truth for the big dataset. They evaluate their model using Change Detection 2014 dataset, and results outperformed other state-of-the-art methods with F-measure of 0.982 with ∼30 fps processing speed. Introduction Moving objects segmentation is important in intelligent video surveillance for a smart city. A common strategy for segmenting moving foreground is to perform background subtraction which is a binary classification problem for each pixel. Several algorithms have been proposed [1]. There is no general robust and efficient approach in dealing with a complex scenario like illumination changes, dynamic background, camouflage, camera movement etc. Considering the video surveillance condition, a robust segmentation method should perform well under the aforementioned situations. With excellent feature representation and data fitting ability, convolutional neural network (CNN) has achieved extraordinary performance in many computer vision tasks. In this Letter, we propose a robust and flexible method for foreground segmentation with encoder and decoder network. We adopt the first two stages of the pre-trained resnet-50 net for the encoder net with multiscale features extraction and concatenation structure at the end of it to extract multiscale feature map from the dataset. And we propose a transposed CNN for the decoder to map the feature map to the foreground probability map. The proposed method produces impressive results on the publicly CDnet2014 dataset. Training samples selection Samples selection for training a scene-specific model is important for dynamic backgrounds. The number of samples has a positive correlation with the motion degree of background, the more dynamic background degree is, the more samples are needed for training. We select more samples containing foreground objects using motion detection strategy in [2] which could select frames with foreground objects automatically. The procedure is illustrated below. Let represent the greyscale image sequence, N is the total number of frames, where h and w denote the height and width, respectively. First, calculate differences of consecutive images to denote the motion, and calculate the summation of all items: (1)Normalise the vector (2)Then calculate the differential of (3)Normalise in the same way as in (2), and then we get . The relevant frames are selected by the threshold method (4)where is the average of vector could be manually set by empirical. means the frame is to be selected. We process video sequences in this way with threshold , and randomly select 200 frames from the results of frame selection. Training with balanced data Balanced number distribution of positive and negative data is important for a supervised training process. In our training network, considering that the number of background pixels is generally out-weight, the number of foreground pixels in most video sequences, which would lead to suboptimal results in binary classification. In our work, we use a weight-penalty strategy to alleviate the imbalance degree of data which would penalise the loss more if the foreground pixel is classified as background with weights for each training ground-truth. For every segmentation map (ground-truth), allocate weights to the positive and negative abiding the following steps. Let denote the flatten segmentation map, calculate the total number of pixels of (5)Calculate the total categories of (6)Calculate the cumulative frequencies of each category (7)The allocated weights are calculated as (8) Encoder network We use the first two stages of resnet-50 to extract low-level features. ResNet is proposed in 2015 to deal with degradation problem with the residual unit. The ResNet structure could cut down the training parameters compared with VGG-16, and could accelerate the speed for training. At the end of it, a multiscale feature extraction module, which is proposed is attached using hybrid dilated convolution with dilation rate {1, 2, 5}, which is similar to the FPM module with dilation rate {4, 8, 16} of FgSegNet_v2 [3] but get avoid of gridding effect [4]. In this module, feature map is processed through five different calculation processes, in which three different scales of the feature are extracted and concatenated, and then the five processed features got concatenated and normalised as the final feature map. The details of the calculation could be referred in Fig. 1. Fig 1Open in figure viewerPowerPoint Multiscale feature extraction module in encoder network Dilation rate in [3] is {4, 8, 16}, as the dilation rate gets bigger, the sampling would be sparse, which would lose many local information and break the consecutive relation among local pixels, this side effect is called gridding effect. The input is RGB images. Assume that the width and height of image is w and h, respectively, then the size of input image is W × H × 3. At the end of the first stage of encoder, the input image is processed with convolutional layer into a W × H × 64 feature map. Then the feature map would be down-sampled to by max-pooling with stride 2. The second stage of ResNet consisted of one convolutional block and two identity blocks all of which are filtered by {64, 64, 256}. Then the feature maps are of size . Put the feature maps into multiscale feature extraction module to get the concatenated feature maps and apply dropout of 0.25 at the end of multiscale feature extraction module to avoid over-fitting. The feature maps are of size as the output of the encoder part. Decoder network Our decoder network contains three 3 × 3 transposed conv layers, each conv layer is followed by normalisation and activation layers. Decoder network also contains three up-sampling layers and one 1 × 1 conv layer to map project the feature map to the image. In the decoder part, one global average pooling from the first stage of ResNet right before max-pooling is multiplied with the output features of the third convolutional layer and then the feature is added up with the original output features of the third convolutional layer, which could combine low-level features and high-level features. Details are depicted in Fig. 2. Fig 2Open in figure viewerPowerPoint Illustration of decoder network We use transposed convolutional layers to project high-level feature maps into lower ones. For example, feature map of shape w × h × 320 could be projected into w × h × 64 using the 3 × 3 transposed convolution operator with stride 1. The first three blocks of the decoder part project high dimension feature maps into lower dimension feature maps in this way. And the last layer is a 1 × 1 convolution with a sigmoid function to generate a probability mask of each pixel' probability of being a foreground pixel. Experiments on change detection 2014 dataset We implement our network using Keras with tensorflow as the backend, and test our work on NVIDIA GTX TITAN X. We train our model using N training frames of each dataset, where N is optional (N = 200 in our experiments). Each frame contains M pixels, and we use a binary cross entropy loss function to compute the distance between the predicted label and the true label, the loss function is defined as (9)where is the true label of the pixel i, and is the predicted value of the pixel i. Our performance evaluation is described in Table 1. Other results are cited in [3] and change detection website [5]. Table 1. Performance comparison on Change Detection 2014 dataset Precision Recall F-Measure ours 0.9844 0.97916 0.98166 FgSegNet_v2 0.9802 0.9780 0.9789 BScGAN 0.9232 0.9983 0.9339 Cascade 0.9506 0.8997 0.9209 DeepBS 0.8845 0.8761 0.0.95 PAWCS 0.7718 0.7857 0.7403 SuBSENSE 0.8128 0.7509 0.7408 GMM 0.6846 0.6025 0.5707 Values in bold signify better quantitative results by us compared with other results Fig. 3 shows the visual comparison between the ground-truth and the foreground segmentation results of our method. Fig 3Open in figure viewerPowerPoint Visual comparison: segmentation map (partially displayed) a Original images b Ground-truth c Foreground segmentation results by our algorithm Conclusion Experiments show that the method proposed in this Letter, is excellent in foreground segmentation performance over other state-of-the-art methods. What is more, it is efficient with processing speed as fast as 16–32 ms per frame (∼30 fps) in our test which is the fastest network structure in this field to my knowledge. References 1Bouwmans, T.: 'Traditional and recent approaches in background modeling for foreground detection: an overview', Comput. Sci. Rev., 2014, 11–12, pp. 31– 66, (https://doi/org/10.1016/j.cosrev.2014.04.001) 2Fu, H.N., Wang, B.Z., Liu, H.Z.: 'Fast background estimation on long video sequence', Electron. Lett., 2019, 55, (16), pp. 888– 890, (https://doi/org/10.1049/el.2019.1178) 3Lim, L.A., Keles, H.Y.: ' Learning multi-scale features for foreground segmentation', eprint arXiv:1808.01477, August 2018, https://doi/org/10.1007/s10044-019-00845-9 4Wang, P., Chen, P., Yuan, Y.: ' Understanding convolution for semantic segmentation', eprint arXiv:1702.08502, February 2017, https://doi/org/10.1109/WACV.2018.00163 5Change Detection website. Available at http://www.changedetection.net Citing Literature Volume56, Issue12June 2020Pages 597-599 FiguresReferencesRelatedInformation