The trade‐off between accuracy and the complexity of real‐time background subtraction

IET Image ProcessingVolume 15, Issue 2 p. 350-368 ORIGINAL RESEARCH PAPEROpen Access The trade-off between accuracy and the complexity of real-time background subtraction Md Alamgir Hossain, orcid.org/0000-0002-6865-6650 Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Republic of KoreaSearch for more papers by this authorVanDung Nguyen, orcid.org/0000-0002-3940-3929 Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Republic of KoreaSearch for more papers by this authorEui-Nam Huh, Corresponding Author johnhuh@khu.ac.kr Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Republic of Korea Correspondence Eui-Nam Huh, Department of Computer Science and Engineering, Kyung Hee University, 1732, Deogyeong-daero, Giheung-gu, Yongin-si, Gyeonggi-do, 17104, Republic of Korea. Email: johnhuh@khu.ac.krSearch for more papers by this author Md Alamgir Hossain, orcid.org/0000-0002-6865-6650 Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Republic of KoreaSearch for more papers by this authorVanDung Nguyen, orcid.org/0000-0002-3940-3929 Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Republic of KoreaSearch for more papers by this authorEui-Nam Huh, Corresponding Author johnhuh@khu.ac.kr Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do, Republic of Korea Correspondence Eui-Nam Huh, Department of Computer Science and Engineering, Kyung Hee University, 1732, Deogyeong-daero, Giheung-gu, Yongin-si, Gyeonggi-do, 17104, Republic of Korea. Email: johnhuh@khu.ac.krSearch for more papers by this author First published: 06 December 2020 https://doi.org/10.1049/ipr2.12026AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Background subtraction used in object detection, tracking and action recognition is a typical method that separates foreground objects from the background. These applications require accuracy and a complexity reduction technique. Some approaches have been proposed to either increase accuracy or decrease complexity. However, the trade-off between increasing accuracy and reducing the complexity of background subtraction is a big challenge. To address this issue, a background subtraction-based real-time moving object-detection approach is proposed. The key contribution in authors' proposal is to use a colour image and a novel colour-gradient blending fused image to achieve accurate background/foreground segmentation. The fused image is a combination of a gradient image and a colour image to correct illumination variations and preserve the edge information. Also, thresholds are adaptively selected based on the dynamic background behaviour to attain a more robust classification system. The proposed model based on real-time and complex videos from the CD-2012 and CD-2014 change detection data sets, and the CMD data set is evaluated. Experimental results indicate that authors' method processes around 43 frames per second and requires six bytes of memory per pixel, which is noticeably more efficient and less complex than other background subtraction methods. 1 INTRODUCTION For many years, background subtraction (BGS) has attracted a lot of attention in computer vision and image processing for its broad array of applications. BGS extracts moving objects from a video. The approach needs to model the background and must capture different types of background to detect moving foreground (FG) objects. A frame-difference BGS approach detects moving objects from the absolute intensity difference between the observed frame and the reference frame. On the other hand, some sample consensus approaches used local binary pattern (LBP) [1], local binary similarity pattern (LBSP) [2, 3], local singular value decomposition binary pattern (SVDBP) [4], or gradient magnitude (GM) [5] in addition to colour to increase accuracy. Although the additional features with colour-based BGS increase accuracy to some extent, the method brings extraneous complexity. Therefore, to solve the complexity-versus-accuracy problem, we utilize a colour-feature and novel colour-gradient blending fused feature for background/foreground segmentation. The new fused feature is created from the colour feature and the gradient feature. There have been many proposed approaches to object detection, all with different goals. We classify object detection methods into two categories: typical object detection methods and moving-object detection methods. The typical object detection approach is the process of segmenting objects, such as humans, cars, trees, and buildings in images or videos. On the other hand, the latter approach classifies objects in motion, such as humans, vehicles, and animals. BGS is mainly used in background reconstruction, moving-object classification, and moving-object detection. The moving object-related applications are in intelligent visual surveillance systems, like road traffic surveillance, video surveillance, and maritime and airport monitoring [6, 7]. Other applications include human–computer interaction, traffic monitoring, real-time gesture recognition, content-based video coding, optical motion capture, object tracking, and activity recognition [6, 8, 9]. The demand for real-time moving-object detection is growing nowadays. Real-time moving-object detection and tracking are promising applications in the smart home, the smart environment, smart cities, robotics, and drones. The Internet of Things, cloud computing, fog computing, and edge computing also use detection services for surveillance [10]. Many sensors have limited memory and processing capabilities. Hence, we need applications that utilize fewer resources and yet still provide accuracy. According to our recent review of moving-object detection, we conclude that although some methods are quite effective in terms of accuracy, their complexity is not acceptable for extensive use. Compared to a shallow learning-based method, an accurate deep learning-based method incurs huge computational costs, and requires a large volume of ground truth data to train the model. Bouwmans and Garcia-Garcia [6] identified the present trends in the BGS approaches in terms of effectiveness, memory usage, and time complexity. Therefore, we propose a BGS-based real-time moving-object detection approach. The proposed method can be applied in limited-computing and low memory-space devices, and is effective and efficient. Moreover, it can work well despite dynamic backgrounds, thermal effects, unstable video, bad weather, and turbulence. There are many challenges in BGS or moving foreground detection, such as illumination variations, light switching, dynamic backgrounds, shadows, motionless moving objects, camouflage, sleeping foreground objects, camera jitter [6], noisy images, inserted background objects [11], and so on. To solve such challenges, many methods have been proposed. A frame-difference method [12, 13] was proposed for background/foreground segmentation that was effective to some extent for non-challenging video cases. Although the method can quickly classify background and FG, it is very sensitive to illumination variations and camera jitter. But, this kind of approach fails when there are either mild or large variations in the background of a video. Later on, the statistical method [14-20] and the codebook method [21] tried to make an illumination-invariant pixel model. Although the accuracy of this method increased compared to the frame-difference method, the method underperforms when dealing with background variations, just like the frame-difference method. Also, the sample consensus-based approaches [2, 22, 23] achieved some success with intensity changes, but the methods sharply decreased in accuracy when there are large, unexpected changes in a video. To work in dynamic backgrounds (either mild or large changes in background), sample consensus-based approaches [2-5, 24, 25] added some pixel descriptors to create an illumination- and shadow-invariant model, and introduced a dynamic background control mechanism. Later on, deep learning-based BGS (DeepBS) [26] exploited two methods [3, 19] to create backgrounds to train and test the model. Although the methods [3, 26] increased performance to some extent, they incurred unnecessary costs. Therefore, it is difficult to exploit these approaches in real-time applications. Some BGS approaches [1, 5, 27, 28] dispense local descriptors, like LBP, LBSP, and GM to get the texture and neighbour information, correct the intensity, and remove shadow. The LBP- or LBSP-based BGS incurs a higher cost than gradient-based BGS. In LBP [1, 27], a local binary pattern emerges from matching each neighbouring pixel with the centre pixel based on a constant threshold. The threshold is some fraction of the centre pixel. Unlike LBP, LBSP [28] encodes the pattern based on the similarity between the centre pixel and the neighbouring pixels. Although LBP and LBSP descriptors are said to be intensity-invariant, they fall in a flat region, as shown in Figure 1, which results in false detections. The window size of the descriptors affects intensity correction. For example, considering the same window size of 5 × 5 (Figure 1), the LBSP pattern shows more variations in frame # 160 than the corresponding patterns in frame # 1 and frame # 83 . Therefore, large block-size descriptors are very sensitive to mild variations in the background. In the same way, LBP and GM also exhibited different patterns and magnitudes for those three frames. On the other hand, in our design, a colour-gradient blending fused image (FI) with a 3 × 3 kernel shows the same values in cases with only the background, the shadow, and the front side of the foreground. The FI can reduce illumination variance. In conclusion, the novel FI can detect all the edges and correct the intensity to some extent at the same time. Deep learning-based typical object-detection approaches, such as the fast region-based convolutional neural network (Fast R-CNN) [29], Faster R-CNN [30], you only look once (YOLO) [31], and Mask R-CNN [32] were proposed for accurate object detection. However, those methods did not mention how to detect moving objects. Alternatively, the traditional image processing-based methods such as Visual Background extractor (ViBe) [23], ViBe + [24], pixel-based adaptive segmenter (PBAS) [5], LOBSTER [2], Self-balanced Sensitivity Segmenter (SuBSENSE) [3], and Sub-super pixel-based Background Subtraction (SBS) [20], and machine learning-based approaches such as the weightless neural network (WNN) [33], the CNN [34], and DeepBS [26] were proposed for moving object segmentation. PBAS and SuBSENSE used pixel features and dynamic background control mechanisms for accurate segmentation. Our proposal enhances PBAS and SuBSENSE using the novel FI. Our method outperforms the existing approaches in terms of accuracy and computational complexity. FIGURE 1Open in figure viewerPowerPoint Colour intensity, local binary pattern (LBP), local binary similarity pattern (LBSP), gradient magnitude (GM), and fused image (FI) are displayed at ( 344 , 410 ) for frame 1 , frame 83 , and frame 160 . Frame 1 , frame 83 , and frame 160 for the specific coordinates have only background, shadow, and the front side of the foreground, respectively In this research, we propose a novel FI-based background subtraction (FBGS) approach. First, we create the FI, which is a combination of both the gradient image and the colour image and which leads to reduced noise while preserving edge information. Afterwards, we fuse the FI and the colour image, model the dynamic background characteristics, update the background samples, and select an adaptive threshold for background/foreground classification. Finally, we do post-processing to discard noise in raw background/foreground segmentation. We list our key contributions as follows: ( 1 ) We design a novel colour-gradient FI, which is a weighted combination of colour and the GM for effective background/foreground classification. ( 2 ) Under different pixel intensity variations, thresholds are effectively selected at runtime to achieve accurate segmentation. ( 3 ) Our system is based on colour, the FI, and an adaptive threshold, which can improve the segmentation accuracy in challenging cases. ( 4 ) We test our system with 54 videos from the CD- 2012 and CD- 2014 change detection data sets, and the CMD data set. Our experimental results demonstrate the performance effectiveness of our proposed system. In the rest of the paper, Section 2 describes other relevant work. The hyperparameters setup, background model, background/foreground classification, background change measurement, dynamic or non-dynamic region detection, blinking pixel level, threshold factor update, background factor update, background model update decision, background diffusion, unstable video detection, and post-processing are presented in Section 3. We describe the parameters settings, the image denosing effects, the importance of FI magnitude, a comparison of methods, and complexity measurements in Section 4, and we discuss the important aspects of our method in Section 5. Conclusions and future work are elaborated on in Section 6. 2 RELATED WORKS From the aspect of high-accuracy performance, many BGS approaches have been proposed. BGS approaches can be classified into basic/simple methods, statistical- and cluster-based methods, sample consensus-based methods, and learning-based methods. Figure 2 depicts the taxonomy of the BGS algorithm. (1) Simple method. A background model is based on two- or three-frame differences, persistent frame differences, averages, medians, and histograms of pixels in a video. In the two-frame difference method [13] and the block-frame difference method [35], an observed video frame is subtracted from a reference background model to detect moving objects. While the absolute difference between two frames extracts some moving objects, it results in false classifications if there are dynamic backgrounds, intensity changes, colour variations, shadows, unstable video, start-up objects, and so on. The method can only detect the boundary edge of evenly coloured intensity or grey intensity objects. In a median-based BGS, the median of each pixel over the temporal direction separates the background from the FG. The temporal median-based BGS fails when each pixel is observed in less than 50 % of the total observations. Spectral- 360 [36] exploited an image formation model and colour spectrum reflectance, and classified the foreground from the video based on full-colour spectrum reflectance feature resemblance, instead of colour/grey intensity similarity in a pixel. (2) Statistical- and cluster-based BGS methods. In the mixture of Gaussians (MOG) model [37], a pixel belongs to more than one Gaussian distribution, whereas Zivkovic and Van Der Heijden [17] used the K-nearest neighbours (KNN) algorithm for BGS, where a pixel belongs to a single cluster. Although MOG and KNN operate well for non-dynamic backgrounds, they fail to segment a moving object from a multi-modal background. In the Gaussian mixture model (GMM) [14], Stauffer and Grimson observed the pixels' intensity behaviour (road and water surface pixels, and the monitor flickering pixel) over time, and found a lot of intensity variations. The variation remained approximately constant for a static environment under a fixed lighting condition. The GMM cannot accurately track an object inserted into a scene, since it does not consider recent pixel behaviour. In addition, the GMM results in a false classification when an object stays static for a long time, and then starts to change position. A Gaussian distribution can be exploited for BGS when there are small, grey-intensity or colour-intensity variations in a pixel, rather than dynamic backgrounds such as rippling water, a plant's leaves swinging, and the like. Wang et al. [19] presented a method named flux tensor with a split Gaussian (FTSG) model, a combination of the spatio-temporal motion detector called a flux tensor and the mixture of a Gaussian distribution for BGS. Unlike MOG, which shares the same fixed number of Gaussian distributions for both foreground and background, FTSG brings one Gaussian distribution for the foreground and one Gaussian mixture for the background. It is known that MOG is not illumination- and shadow-invariant. Also, the flux tensor cannot detect soft motion. Instead of the per-pixel similarity measure, Chen et al. [20] measured a superpixel-by-superpixel analogy and called it SBS. The superpixel was constructed using a simple linear iterative clustering (SLIC) algorithm, and then, the superpixel was subsequently clustered into a sub-superpixel with the help of a K-means algorithm. Moreover, the process of intensity correction could not be more effective, because intensity varies randomly in reality. (3) Sample consensus-based method. At first, Wang and Suter [22] proposed the novel concept of a sample consensus (SACON) for BGS. Later on, Barnich and Van Droogenbroeck presented ViBe [23] to classify the background or FG. Like SACON, ViBe is also a sample consensus-based BGS approach. Unlike SACON, which updates blob samples and pixel samples simultaneously, ViBe updates only pixel samples randomly over time, based on a conservative update policy. In the conservative update policy, the foreground will never be considered as background in the whole BGS process. Therefore, if moving objects stop, the objects will not be background objects anymore. Afterwards, Van Droogenbroeck et al. improved ViBe, naming it ViBe + [24], introducing an updating factor of 1 / 5 (because of large global motion due to unstable video) instead of one random selection out of 20 pixels (background sample), and added a new post-processing operation. Unlike ViBe, St. Charles et al. utilized the LBSP descriptor and colour feature in the LOBSTER [2] and SuBSENSE [3] methods. Both ViBe and LOBSTER create the same background modelling policy, diffusion mechanism, and median filter-based post-processing. In the same way, Hofmann et al. [5] also came across a sample consensus-based BGS called the pixel-based adaptive segmenter. For the first time, PBAS created a dynamic background modelling mechanism after realizing the essence of updating stored background samples and thresholds for the dynamic background. Therefore, PBAS introduced two parameters: the decision threshold, which was updated dynamically at a fixed scale, and a learning rate calculating the degree of background dynamics in order to update background at runtime. An extension of PBAS introduced by Tiefenbacher et al. [25] used a proportional-integral-derivative (PID) regulator to manage the decision threshold and to change the update rate variables more effectively. In 2015, St. Charles et al. improved their previous LOBSTER method by introducing intra-LBSB, inter-LBSB, blind update (the background could be updated by both foreground pixel and background pixel), a dynamic background modelling mechanism like PBAS, and adaptive post-processing. The goal of this research work is to detect moving objects on a dynamic environment like PBAS and SuBSENSE. To clearly explain the difference between PBAS, SuBSENSE, and our FBGS architecture with classic dynamic-update-based methods, we present the key performance indicators used in each approach and the corresponding operations in Table 1. In 2016, to make a shadow- and illumination-invariant BGS system, Guo et al. [4] converted SVD of an image block (normalized SVD), and finally encoded the LBP pattern of the normalized SVD. Although Guo et al. [4] claimed that SVDBP was more illumination- and shadow-invariant than LBP, the performance of this approach was not as good as the LBSP-based SuBSENSE [3]. (4) Learning-based method. A self-organizing method called spatially coherent self-organizing background subtraction (SC-SOBS) [38] automatically creates a neural network model for background/foreground segmentation. SC-SOBS was proposed to work in a challenging environment, such as a dynamic background, with slow illumination changes, and with camouflage problems. In 2014, the WNN [33] was proposed to adapt to random behaviour in the background and to control the effectiveness of the BGS system using colour intensity history. Braham et al. [34] proposed a CNN-based BGS that matched patchwise background images created by the median-based method [39]. Babaee et al. [26] developed DeepBS, and DeepBS matched the image patch of the background frame generated by the combination of SuBSENSE [3] and FTSG [19], rather than only the frame-difference BGS in the previous approach [34]. The DeepBS method is not completely error-free, because a portion of the image patch could be foreground and background at the same time. Also, a background image created by the other BGS approaches is not completely accurate. Moreover, DeepBS is not self-sufficient, since it depends on two BGS approaches (SuBSENSE and FTSG). Also, FTSG is a fusion of a flux tensor-based BGS and a Gaussian distribution-based BGS. Furthermore, the method shares a concept similar to the previous CNN-based approach. FIGURE 2Open in figure viewerPowerPoint Taxonomy of background subtraction approach TABLE 1. Comparison of PBAS, SuBSENSE, and the proposed FBGS Key performance indicator Method FE1 FE2 background model TH Operation PBAS Colour GM Dynamic Dynamic The segmentation decision is only based on a weighted fusion of colour and gradient magnitude. In addition, in PBAS, the background and the threshold, respectively, are updated according to a conservative policy or the minimal distance between an observed pixel and its stored background pixel. SuBS. Colour LBSP Dynamic Dynamic The segmentation decision is based on colour and LBSP. Moreover, SuBSENSE updates the background and the threshold according to a blind policy and a threshold factor, respectively. The threshold factor is determined not only based on the minimal distance between an observed pixel and its stored background pixel, like PBAS, but also from the noise of the raw segmentation of the previous frame. FBGS Colour FI Dynamic Dynamic FBGS effectively uses colour, the FI, and a fusion of colour and the FI for the segmentation decision. Also, like SuBSENSE, FBGS uses a blind policy and a threshold factor to update the background and threshold, respectively. But under different pixel intensity variations, the thresholds are effectively selected at runtime to achieve accurate segmentation. Abbreviations: BG, background; FBGS, fused image-based background subtraction; FE1, Feature 1; FE2, Feature 2; FI, fused image; GM, gradient magnitude; LBSP, local binary similarity pattern; PBAS, pixel-based adaptive segmenter; SuBSENSE, self-balanced sensitivity segmenter; TH, threshold 3 METHOD PRELIMINARIES The sample consensus-based method is a pixel-by-pixel background/foreground classification method. Hereinafter, we define background samples to represent background pixels. In sample consensus-based BGS, neighbours of each pixel are separately stored as background samples from the first video frame. The background samples are updated based on the background condition over time. Then, an observed pixel from the next video sequences is segmented as an foreground if a certain number of background pixels do not match the observed pixel. The proposed FBGS method uses our newly created colour-gradient blending FI and a colour image to extract the foreground from the BG. We define notations in Table 2. According to LOBSTER [2], SuBSENSE [3], PBAS [5], ViBe [23], and ViBe + [24], these hyperparameters are found empirically. The best values of these hyperparameters are used to achieve the best performance results. Figure 3 depicts the FBGS method. The FBGS model has 13 blocks, as follows. (1) FI creation. The FI is a combination of a colour image and a gradient image. The gradient image is defined as an image that takes a GM of a colour intensity image for each pixel. The GM shows a directional change in the colour intensity of a colour-intensity image. The specific percentage of colour/grey intensity and GM are combined to create the new FI (see Section 3.1). (2) Background model. The background model is created from the background samples. The background sample is the neighbour pixels of a pixel for which the background model is initialized. Two background samples are created for each pixel from the first colour/grey image and the FI frame separately (see Section 3.2). (3) background/foreground classification. The classification starts from the second frame of the video sequence based on similarity/dissimilarity of the pixel's features. The colour/grey intensity and fused magnitude are the features of each pixel (see Section 3.3). (4) Background change measurement. The minimum distance between a pixel and its background samples is estimated in order to measure the background change, which is used to update the threshold factor and background factor (see Section 3.4). (5) Dynamic or non-dynamic region detection. Tree leaves rustling, a fountain, light fluctuations, and the like constitute a dynamic background; otherwise, the background is static (see Section 3.5). (6) Blinking pixel level. A blinking pixel is defined as a pixel that switches between foreground and BG. The number of blinking pixels is estimated from raw segmentation. This level is used to decide the degree of change in the threshold factor and background factor (see Section 3.6). (7) Threshold factor update.The threshold factor is updated over time based on a minimal distance between a pixel and its background samples. This factor is utilized to compute the decision thresholds (see Section 3.7). (8) background factor update. The background factor is used to calculate the probability of a background sample update and the probability of diffusion (see Section 3.8). (9) Decision threshold. The decision threshold determines background or foreground pixels in the background/foreground classification (see Section 3.9). (10) background model update decision. The background samples need to be updated over time. The detected background pixel replaces the background sample based on the probability of pixel replacement (see Section 3.10). (11) Background diffusion. In addition to background sample replacement, the neighbouring pixels' background samples are replaced with the detected background pixels based on the probability of a background update, the ghost artefacts, and the background condition (dynamic or static) (see Section 3.11). (12) Unstable video detection. An unstable video can be caused by a non-static camera. The unstable video replaces the background sample with background pixels more frequently, and increases the threshold more sharply (see Section 3.12). (13) Post-processing. Post-processing discards noise (isolated foreground pixels) and fills the unfilled region of objects detected during background/foreground classification (see Section 3.13). TABLE 2. Description of variables and notations Variable/notation Value Description/Definition x – The row number of a video frame y – The column number of a video frame I – The colour/intensity an image frame F – The fused image frame K x – The horizontal convolution kernel K y – The vertical convolution kernel # c o u n t – The number of matching similarities ∨ – The logical OR operator L 1 ( A , B ) – The Manhattan distance between A and B; let the intensity of a pixel = A , and the pixel's background sample intensity or the pixel's intensity of the previous frame = B L 1 min ( A , B ) – The minimum Manhattan distance between A and B T I ( x , y ) – The decision threshold of a colour/grey image pixel T F ( x , y ) – The decision threshold of a fused image pixel H ( x , y ) – The threshold factor; initially, H ( x , y ) = 1 S ( x , y ) – The stable or unstable background region indicator; initially, S ( x , y ) = 0 ¬ – The complement of S ( x , y ) G ( x , y ) – The ghost artefacts B ( x , y ) – The blinking pixel G m i n 0.995 The minimum ghost artefacts G m a x 0.010 The maximum ghost artefacts α 25 The short-term moving average weight factor β 100 The long-term moving average weight factor R d m 3 The minimum segmentation fault R m 0.1 The minimum unstable ratio of the background region W i c r e 1 The degree of increment in blinking pixel approximation W d c r e 0.1 The degree of decrement in blinking pixel approximation Z 0.01 The threshold factor step size U i c r e 0.50 The degree of the background factor increment U d c r e 0.25 The degree of the background factor decrement γ 0.60 The weight factor to fuse colour and normalized gradient pixels N 49 The number of background samples t 1 , 3 The fusion weight factor of colour/grey image pixel and