Background subtraction in dynamic scenes using the dynamic principal component analysis

IET Image ProcessingVolume 14, Issue 2 p. 245-255 Research ArticleFree Access Background subtraction in dynamic scenes using the dynamic principal component analysis Achraf Djerida, Achraf Djerida Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of ChinaSearch for more papers by this authorZhonghua Zhao, Corresponding Author Zhonghua Zhao zhaozh@sjtu.edu.cn Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of ChinaSearch for more papers by this authorJiankang Zhao, Jiankang Zhao Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of ChinaSearch for more papers by this author Achraf Djerida, Achraf Djerida Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of ChinaSearch for more papers by this authorZhonghua Zhao, Corresponding Author Zhonghua Zhao zhaozh@sjtu.edu.cn Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of ChinaSearch for more papers by this authorJiankang Zhao, Jiankang Zhao Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of ChinaSearch for more papers by this author First published: 01 February 2020 https://doi.org/10.1049/iet-ipr.2018.6095AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This study presents a foreground detection method capable of robustly estimating the background under the presence of dynamic effects. The key contribution of this study is the use of the dynamic principal component analysis to model the serial correlation between successive frames and construct a robust pixel-based background model. The frames are normalised in hue, saturation and value colour space to reduce the effect of illumination changes. To restrict the background model, kernel density estimation is used to identify the distribution of the background time-lagged data matrix and then confidence interval limits are used to determine the corresponding detection thresholds. The foreground is detected using background subtraction. This method is tested on several common sequences such as CDnet 2014, ETSI 2014 and MULTIVISION 2013. The authors also hold comparisons based on quantitative metrics with several state-of-the-art methods. Experimental results show that their method outperforms some state-of-the-art methods and has comparable performance with some depth-based methods. 1 Introduction Background subtraction is the process of separating a static environment (background) from a moving structure (foreground) [1]. Recently, there are numerous applications of this topic such as on surveillance and human–machine interaction [2]. A foreground detection algorithm for real-time multimedia communication systems is proposed in [3]. It avoids computationally heavy operations to make it suitable for practical applications. To boost the performance of surveillance segmentation applications, an evaluation method that takes into account detection failures and false alarms is proposed in [4]. The different proposed criteria can help the user to get the best algorithm. In [5], a machine vision algorithm, which is capable of detecting and counting fish is developed. Similar applications take place in robotics as in [6], where a visual system was developed to track moving objects. So far, the background estimation problem has been tackled with a variety of methods [2, 7]. A simple estimation algorithm for stationary background is proposed in [8] based on running average algorithms. Although it has good computation speed, it breaks down under dynamic scenes. To detect people and interpret their behaviour, a real-time system was developed in [9]. It achieves good performance in terms of the detection accuracy; however, it is based on the assumption that the scene is less dynamic than the user. A texture-based method was used in [10] to detect moving objects. It models each pixel as a group of adaptive local histograms using a circular region. The method requires stationary camera, and it has many parameters, which can limit its performance on practical applications. To gain from the immunity of local texture features to illumination changes, photometric invariant colour measurement is proposed in [11], and it shows a robust performance on both rich texture and uniform regions. To acquire a robust background model, a combination between principal component analysis (PCA) and linear discriminative analysis was proposed in [12]. This method was motivated by the supervision characteristic of linear discriminant analysis over PCA. A wavelet multi-scale transform is used in [13] to separate the background and foreground while suppressing shadow effects. It has many advantages such as automatic detection and its low computation cost. To overcome the effect of dynamic background, a spatio-temporal method was proposed in [14] based on a space–time patch. It was motivated by its ability to represent both the motion and appearance. In addition to colour features, depth features have been introduced on many algorithms as in [15] provided that the background and foreground have different depths. In this paper, we propose a novel framework for background estimation under dynamic effects using the dynamic PCA. The work is motivated by the idea of modelling a sequence of images as an output of a dynamic system. This leads to taking into account the serial correlation between the successive frames instead of assuming samples independence. PCA-based foreground detection methods show promising performance on many applications [2]; however, since PCA ignores the time correlation between samples, it initiates false estimations when the time correlation between samples cannot be ignored. In this work, we use the dynamic PCA to model the dynamic properties between successive frames to construct a robust background model. To reduce the effect of luminance variation, we normalise the images on the hue, saturation and value (HSV) colour space. Kernel density estimation is used to model the distribution of the time-lagged data matrix and foreground detection is acquired using a background subtraction scheme. To validate our method, we hold experiments on several recent sequences and with several state-of-the-art methods. 2 Related work Background subtraction is usually done in two steps: background modelling and foreground extraction [16]. So far, this topic has been exploited widely due to its impact on different applications [17], which include advanced statistical, fuzzy, robust subspace and transform domain models [2]. Statistical models have an advantage toward illumination effects and dynamic backgrounds [2]. A real-time video segmentation algorithm based on Student's t has been used in [18] to create a mixture model. A recursive filter has been used to update the model, so that its computation cost is improved. To detect moving objects accurately in complex scenes, an unsupervised method based on Markov random field is proposed in [19]. Different features have been fused such as colour, locality and temporal coherence and different parametric and non-parametric methods were used to learn their distribution. To make a foreground detection algorithm adaptable to multiple environments, a samples based background estimation method is proposed in [20]. It uses different aggregation policies to select model values, and it does not need a fine-tuning algorithm. A non-parametric background model was developed in [21] to make efficient use of past values. Its detection thresholds are extended to dynamic per-pixel state variables together with the notion of dynamic controllers. A mixture of Gaussians is one of the common techniques in the field of foreground detection as in [22], where a constrained mixture of Gaussians is developed to learn the variance of the pixels as a part from the background model. Fuzzy models have been incorporated on many background estimation methods to tackle their uncertainties [2]. A Type-2 fuzzy mixture of Gaussians model is proposed in [23] to take into account the uncertainties resulted from dynamic effects. This scheme was shown to be effective against waving trees and water rippling. To attenuate colour variations, a fuzzy colour histogram was developed in [24]. The background pixels are identified based on a similarity measure. Similarly, a fuzzy logic model was developed in [25] based on scene parameters. It provides an automatic way for foreground detection, and it does not need user intervention. To tackle the effect of dynamic backgrounds, a fuzzy membership transformation was developed in [26] to create a rich fuzzy vector. The fuzzy statistical textural features are used as the basis for foreground detection. Experimental results show that these features can be applied to other vision-based applications. Robust subspace methods aim at the decomposition of background and foreground via a robust subspace model [2]. To solve the background estimation problem under multiple dynamic effects, the background modelling is seen as pursuing subspaces within the video bricks as in [27], and an autoregressive (ARX) moving average model is used to characterise their appearance consistency. To make the algorithm robust to disturbances and scene changes, the subspaces are incrementally updated. Another way of factoring the background and foreground can be accomplished based on near-separable non-negative matrix factorisation as in [28], where the developed method has many advantages such as scalability and noise tolerance. Since robust PCA is the crux of many background subtraction methods, many related topics have been questioned in the literature. In [29], a study that concerns the recovery of each component when the data matrix is the superposition of a low-rank component and a sparse component. Similarly in [30] where the recovery of low-rank matrix from a high-dimensional matrix becomes a challenge when small noise and gross sparse errors exist. Transform-based methods aim at separating the background and foreground on a different domain [2]. Representing dynamic background as a dynamic system, which has frequency responses can be a good model as shown in [31]. Under this case, the present frame is assumed to be correlated with past frames. Similarly, a frequency decomposition model that explicitly represents the scene dynamics is proposed in [32]. It can recognise some dynamics such as waves and plants motions. To identify such motion, frequency coefficients are calculated for each pixel in moving windows. To take into account both the accuracy and stability of foreground segmentation, a Gaussian mixture model is used with Walsh transform in [33] to characterise the background pixels. The Hadamard transform is used in [34] instead of the discrete cosine transform to increase the estimation speed. Experimental results record ten times faster than some of the related methods. Recently, the work of background subtraction is directed more toward developing robust algorithms against multiple dynamic effects. In [17], an algorithm was developed to be adapted to different motion speeds using a neighbour-based intensity correction, which compares the current and previous frames to determine the adaptation decision. Foreground detection is accomplished using a threshold detection scheme based on the Otsu method. Although it shows its effectiveness under many dynamic effects, the computation burden and sensitivity toward repetitive motions decrease its performance. To tackle fast dynamic backgrounds, a region-based approach is used to generate the background model as in [35]. Experimental results show that the inclusion of neighbouring pixels improves the mixture of Gaussians model greatly. Although this scheme is suitable for dynamic scenes, many related topics still open such as the determination of the optimal region size. To achieve effective change detection, variable weights are used to scale the different samples as in [36], where it can reduce the false updating. To handle the situation where different challenges are present, a robust encoder–decoder neural network was developed in [37] to extract multi-scale features which can characterise the background robustly. A mapping is then learnt using convolutional neural networks on the decoder part. Once the model is trained on few samples, it can be used to segment the foreground on complex situations. To improve the accuracy of the segmented foreground and provide accurate ground-truth results for video surveillance tasks, a multi-resolution convolutional neural network with cascaded architecture was used in [38] to model the background. As redundant information of background and foreground is present through subsequent frames, only few samples are needed to initialise the model. To avoid parameter tuning and feature engineering, a foreground segmentation method based on convolutional neural networks is developed in [39], where the network parameters are learnt directly from the ground-truth data. To reduce the computation cost while maintaining good accuracy, a background model based on sub-superpixels is developed in [40]. As the method uses simple linear clustering techniques for model construction, it leads to more efficient performance. To boost the accuracy of both the background and foreground, a background model based on fuzzy histograms is developed in [41]. In this model, the temporal characteristics of the pixels are described using the fuzzy c-means algorithm, and an adaptive threshold is used to segment the foreground. Table 1 summarises the common background estimation categories and their basic ideas based on the work done in [2]. Table 1. Summary of common background subtraction methods Category Examples Key idea statistical models [42, 43] represent the background using a traditional statistical model (for example, gaussian mixture models (GMM)) advanced statistical models [18, 19] represent the background using an advanced statistical model (for example, student-t mixture model (STMM)) fuzzy models [24, 25] introduce fuzzy concepts on the estimation process (for example, type-2 fuzzy mixture of gaussians (T2F-MOG)) robust subspace models [28, 29] separate the background and foreground using a robust subspace method (for example RPCA) transform domains [31, 32] separate the background and foreground on a different domain (for example Fast Fourier transform) 3 Background subtraction using dynamic PCA 3.1 Method overview The proposed method aims at estimating the background model, which can be used to find the foreground based on a modal violation scheme. The images are modelled as an autocorrelation process to cope with the evolution of new images, which usually have a relationship with the previous samples. This formulation is motivated by the similarity to dynamic systems. The dynamic PCA is applied to the model initialisation phase to provide a mapping for each pixel. On the new feature space, statistical thresholds are developed to restrict the background. To make the proposed method robust toward illumination changes, the HSV colour space is used to normalise the images’ luminance as a preprocessing phase. 3.2 Establishment of the autocorrelation model Background estimation can be a challenging task when dynamic effects are present. The modelling of a static background is an effective method to detect any foreground provided that the estimated model is accurate [35]. In this section, we show how the dynamic PCA model can be a feasible solution to tackle such dynamic effects. Different statistical based methods have been applied to estimate the background models such as PCA, where they show great potential in separating the background and foreground pixels [2]. Since PCA does not take into account the series correlation, the estimated model can diverge under dynamic conditions [44]. Motivated by this fact, we use dynamic PCA to estimate the background and foreground pixels. Let us assume that the data matrix , which represents n frames with m variables, is given by (1)where represents an intensity value. We assume that the process is stationary [45], which means that the joint cumulative function of the following finite-order distributions are the same for every N, , , and time shift : (2)To model the autocorrelation between current and previous observations, the ARX models can be used for single-input–single-output system as (3)where , represent the output and input of the dynamic system at the instant t. and are process coefficients. is a white noise process and h is a lag value. The current output value does not depend only on the current input but it also depends on the past h input and output values. Measurements corresponding to (3) can be put into a matrix form as (4)where is vector holding the process coefficients and is a vector holding the noise processes. can be formulated as (5)The matrix reveals that the first column is linearly related to the other columns. Applying PCA to (5) is what is referred to as the dynamic PCA [46]. It involves two steps: generating the trajectory matrix and extracting the loading matrix using PCA. The trajectory matrix can be formulated based on the data matrix in (1) as [44] (6)where means the ith row of the matrix X. To extract the loading matrix P, we first centralise the trajectory matrix using its mean defined as (7)Owing to the stationarity property defined by (2) (8)Since practically these conditions may not be fully satisfied, any violation of this property induces false detections. Similarly to PCA, we search for the loading matrix P, which is orthogonal and its variances should be maximal in the directions of the new data T given by (9)where is the centralised trajectory matrix. To find the vectors of P, an optimisation problem has to be solved. To maximise the variance of the scores, the following expression is maximised: (10)where corresponds to the jth component of the score matrix T, corresponds to the jth component of the loading matrix P and is a vector from the trajectory matrix . Solving this problem using Lagrange multipliers leads to [47] (11)This is a classical eigenvalue problem, where , represents its eigenvalues and represents its eigenvectors. Fig. 1 shows the basic steps for the construction of the dynamic PCA model during the training and application phases. Fig. 1Open in figure viewerPowerPoint Basic steps to construct the dynamic PCA model 3.3 Background estimation using the dynamic PCA model In this section, we use the dynamic PCA to construct a model for the background. While the dynamic PCA model can estimate dynamic backgrounds, it fails against changing illumination effects. As the illumination variation changes the colour properties of the pixels, false detections are generated due to the violation of the stationarity property defined by (2). To solve this problem, there are mainly two methods: Threshold adaptation: In this method, the thresholds are updated by comparing the current and previous images and based on a change detector the background is updated. This method can be a good solution for slow variations; however, it can get into trouble under great changes. Model normalisation: This method looks for an illumination invariant space where the background model is independent of these changes. In this work, we adopt this scheme since it can be more reliable under high illumination effects. To reduce the illumination effects, we transform the red, green and blue (RGB) image to the HSV colour space, where the normalisation takes place. The reason for selecting this colour space for normalisation is owed to the characteristics of its channels, which can lead to the separation of the luminance from the chrominance (colour) [48]. While the hue of a colour determines which pure colour it resembles, its saturation determines the amount of grey, which is contained in colour. The value channel measures the intensity of the colour. Consider a test image I in the RGB space, and we first convert it to the HSV space HSV (I) to calculate the mean of the value channel as (12)where V represents the value channel and and are the number of rows and columns. To normalise images, we centralise the value channel using the following equation: (13)where represents the desired mean value and is a constant used to modify the contrast of the image. The value of is determined during the background initialisation and all the images are normalised using (13). Fig. 2 shows the effects of and on a reference image after converting it back to the RGB colour space. Fig. 2Open in figure viewerPowerPoint Effect of the HSV colour space coefficients (a) = reference, , (b) = reference, , (c) , , (d) , Let the background initialisation stage runs over frames. The value of is determined as the average value of the calculated mean using (12). To make the estimation process more robust, we assign for each pixel with image coordinates a dynamic PCA model generated by (6) and (9). The X matrix defined by (1) is used to represent the values of the RGB channels as (14)where represents the intensity of the channel q at the frame p. Although PCA is used generally for dimension reduction, it can be used to decorrelate the data and provide a monitoring statistic to characterise the background [46]. This is in coincidence with many works, where the introduction of the PCA results in better performance or at least the same error as in [49, 50]. The data matrix in (14) is transformed into the centralised trajectory matrix by (6) (15)Clearly, for the processing of a new frame, we need to save the last h samples in the delay vector as (16)After the centralisation of the trajectory matrix, the score space is generated by (9) as (17)where is the loading matrix for pixel . To determine the background model on the score space, we use the norm of the score vectors as a statistical measure as (18)where l denotes the frame number. To restrict the background region in the score space, upper and lower thresholds are determined. Since assuming that the norm statistic follows Gaussian distribution can be violated practically, we use kernel density estimation to find its distribution as a sum of kernels [51]: (19)where s is a data point, which we want to find its density value , is the norm of the pixel at the frame l, K is a kernel function (normal density function), is a smoothing parameter and n represents the total number of initialisation frames. To find the confidence interval thresholds, we calculate the cumulative distribution function as (20)The lower and upper thresholds and , for a given lower and upper confidence intervals and , can be given as (21)Fig. 3 shows the basic steps for the construction of the background model. Fig. 3Open in figure viewerPowerPoint Basic steps for the construction of the background model 3.4 Foreground detection The foreground can be extracted by subtracting the estimated background from the current image. Let us assume that the parameters and are estimated during the background initialisation phase. For a new frame, we update the delay vector for each pixel using (16), and we calculate its norm statistic, which is defined by (18). Each pixel is classified based on the following constraints. For pixel (22) 4 Experimental results and discussion 4.1 Datasets and evaluation criteria In this section, we use 13 videos to evaluate the proposed method with comparisons with the state-of-the-art methods. We select videos from four recent datasets: the CDNet2014 [52], ETSI [53], MULTIVISION [54] and UR fall detection [55]. Fig. 4 reports some characteristics of these videos: (i) ETSI: It is one of the first published RGB depth (RGBD) benchmarks for evaluating background estimation methods under a variety of effects [53]. From this dataset, we select four videos from four categories: GenSeq1 from the illumination changes, DCamSeq2 from depth camouflage, TopViewLab2 from the out of range sensor and ColorCam2 from colour Camouflage. Fig. 4Open in figure viewerPowerPoint Details of the different videos (ii) MULTIVISION: It is characterised by the inclusion of the depth information in addition to the RGB images [54]. From this dataset, we select the ChairBox sequence, which includes illumination effects. (iii) UR fall detection: It contains different fall scenarios with both RGB and depth images. We select Fall01cam1 sequence, which contains shadow effects. (iv) CDnet 2014: It is an expansion to CDnet 2012, and it is created to evaluate change and motion detection approaches [52]. We select seven videos from five categories: boats and overpass from dynamic background, copy machine from shadows, boulevard from camera jitter, sofa from intermittent motion and office and pedestrians from baseline. There exist many criteria to evaluate the quality of the estimated background [56], and in this work we use three criteria [52] which are the recall, precision and F-measure. The precision measures the percentage of the truly extracted foreground pixels to the total number of extracted foreground pixels. The recall can be defined as a ratio between the number of truly extracted pixels and the actual number of foreground pixels. The F-measure is a compromise between the recall and precision [56]. 4.2 Effect of the proposed algorithm parameters In this section, we determine the values of the lag value h and the contrast coefficient based on their effect on the recall and F-measure. Since the recall measures the detection rate and the F-measure takes into account both the recall and precision measures, both of them can give a good evaluation of the quality of estimation. In determining the lag value, the contrast coefficient is held constant, and then the reverse is true for the contrast coefficient. In the literature, there is no optimal way to determine the lag value h [46]; however, usually some heuristic methods are used. Common methods include the use of 1 or 2 lagging samples as in fault detection systems [46] or making it a fraction (25%) from the total number of observation [44]. In this work, we determine its value as the one, which results in the best performance based on the recall and F-measure. For this reason, we choose the pedestrians and overpass as test sequences as shown in Fig. 5. We can see that both the recall and F-measure increase as the lag value increases until value 2, where the recall continues increasing while the F-measure starts dropping. On both sequences, we see that the detection rates rise by around 10% as the number of lagging samples increases from 1 to 2 (while the F-measure increases) which guarantees the balance between the detection and precision accuracies. Therefore, the lag value is fixed to 2 for all the experiments. Fig. 5Open in figure viewerPowerPoint Effect of the lag value and contrast coefficient on some sequences (a) Effect of the lag value as applied to pedestrians,(b) Effect of the lag value asapplied to Overpass, (c) Effect of thecontrast coefficient as applied to pedestrians,(d) Effect of the contrastcoefficient as applied to overpass The second parameter is the contrast coefficient . Its role is to set up a reference brightness as shown in Fig. 2. Similarly, to the lag value, we use the recall and F-measure to determine its value. Fig. 5 shows the corresponding performance for both the pedestrians and overpass. As the contrast coefficient increases, the recall increases on both datasets. However, the F-measure increases until five, where it starts to decrease rapidly. The increase of the contrast coefficient can improve the algorithm performance provided that it is less than its breaking point which is in this case 5. Therefore, the contrast coefficient is fixed to 4.5 just before the breaking point to avoid possible degradations. 4.3 Qualitative and quantitative results In this section, we hold experiments on different datasets with comparisons with some recent state-of-the-art methods. The parameters of the algorithm are fixed for all the datasets as follows: the number of frames used for background initialisation , the lag value and the contrast coefficient . To make fair comparisons, we compare our algorithm with some methods, which have been applied to these datasets and reported by their authors. The resulted images of the state-of-the-art methods for the CDnet 2014 are reported in the dataset website [52] and for the rest of datasets are provided on the scene background modeling (SBM)-RGBD website [57]. To evaluate the three criteria for all the methods, we use the provided scripts in [52]. 4.3.1 Results on CDnet2014 sequences The proposed algorithm is compared with four state-of-the-art methods: WeSamBE [36], boosted Gaussian mixture mo