Dense optical flow based background subtraction technique for object segmentation in moving camera environment

IET Image ProcessingVolume 14, Issue 14 p. 3393-3404 Research ArticleFree Access Dense optical flow based background subtraction technique for object segmentation in moving camera environment Arati Kushwaha, Arati Kushwaha Department of Electronics and Communication, University of Allahabad, Allahabad, IndiaSearch for more papers by this authorAshish Khare, Corresponding Author Ashish Khare khare@allduniv.ac.in Department of Electronics and Communication, University of Allahabad, Allahabad, IndiaSearch for more papers by this authorOm Prakash, Om Prakash Department of Computer Science and Engineering, H.N.B. Garhwal University, Srinagar (Garhwal), IndiaSearch for more papers by this authorManish Khare, Manish Khare orcid.org/0000-0002-2296-2732 Dhirubhai Ambani Institute of Information and Communication Technology, Gandhinagar, IndiaSearch for more papers by this author Arati Kushwaha, Arati Kushwaha Department of Electronics and Communication, University of Allahabad, Allahabad, IndiaSearch for more papers by this authorAshish Khare, Corresponding Author Ashish Khare khare@allduniv.ac.in Department of Electronics and Communication, University of Allahabad, Allahabad, IndiaSearch for more papers by this authorOm Prakash, Om Prakash Department of Computer Science and Engineering, H.N.B. Garhwal University, Srinagar (Garhwal), IndiaSearch for more papers by this authorManish Khare, Manish Khare orcid.org/0000-0002-2296-2732 Dhirubhai Ambani Institute of Information and Communication Technology, Gandhinagar, IndiaSearch for more papers by this author First published: 29 October 2020 https://doi.org/10.1049/iet-ipr.2019.0960Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Segmentation of moving object in video with moving background is a challenging problem and it becomes more difficult with varying illumination. The authors propose a dense optical flow-based background subtraction technique for object segmentation. The proposed technique is fast and reliable for segmentation of moving objects in realistic unconstrained videos. In the proposed work, they stabilise the camera motion by computing homography matrix, then they perform statistical background modelling using single Gaussian background modelling approach. Moving pixels are identified using dense optical flow in the background modelled scenario. The dense optical flow provides motion information of each pixel between consecutive frames, therefore for moving pixel identification they compute motion flow vector of each pixel between consecutive frames. To distinguish between foreground and background pixels, they labelled each pixel and thresholding the magnitude of motion flow vector identifies the moving pixels. The effectiveness of the proposed algorithm has been evaluated both qualitatively and quantitatively. The proposed algorithm has been evaluated on several realistic videos of different complex conditions. To assess the performance of the proposed work, the authors compared their algorithm with other state-of-art methods and found that the proposed method outperforms the other methods. 1 Introduction In past two decades, computer vision community has been working on different areas like robotics, automated surveillance, vehicular traffic analysis, human–computer interaction, human activity recognition and so on, mainly to deal with safety and security [1] concerns. Moving object segmentation is one of the important steps in computer vision applications. Segmentation of moving object by moving camera such as pan-tilt-zoom (PTZ) camera or hand-held camera is a challenging and difficult task due to varying background along with the motion of camera. Moving object segmentation is a process of detection of intended object and/or detection of foreground pixels in video frames. The segmentation of moving object is a difficult task [2, 3] due to the following reasons: (i) 2D scene is projection of 3D world. (ii) Presence of noise and blur. (iii) Complex motion of object. (iv) Camera motion. (v) Object occlusion. (vi) Varying object shape from frame to frame. (vii) Dynamic background. (viii) Varying illumination condition. In the present work, we propose a dense flow based moving object segmentation method in moving camera environment. The segmentation of moving object was commonly performed by the background subtraction techniques [4–11]. The background subtraction techniques perform subtraction between background and current frame. These methods performed poor segmentation in case of slow motion of object, abrupt change in illumination, complex background which were not well suited for realistic unconstrained videos. To address these problems, we propose a dense optical flow based background subtraction approach in moving camera environment. We integrated single Gaussian based background subtraction technique with dense optical flow method to compute motion flow vector. The main motivation behind use of dense optical flow [12–15] for computation of motion flow vector is that background pixels have negligible motion as compared to foreground pixels [16]. Firstly, camera motion was estimated and then background motion has been stabilised to get frame sequence. These frame sequences could be treated as frames in videos recorded with a static camera. For videos with varying illumination and slow object motion, false labelling problem is the major issue in object segmentation [16]. To overcome the false labelling problem, we use motion vector of each pixel. Since, in stabilised camera motion and background modelling, each background pixel have negligible motion than the moving pixels. The proposed object segmentation approach consists of following subtasks: (i) Stabilisation of camera motion by computing overlapped region between current frame and background. (ii) Background modelling using single Gaussian modelling approach with varying learning rate. (iii) Computation of motion flow vector of each pixel, using dense optical flow between current frame and background. (iv) Segmentation of moving object by subtracting the background frame from current frame followed by thresholding the magnitude of motion flow vectors of each frame. The performance of proposed method is evaluated and compared with other state-of-the-art methods. The proposed method is compared with the nine well-established methods for moving object segmentation in moving camera environment. The methods used in comparison are as follows: (i) Moving object segmentation based on scene conditional background updation approach, proposed by Yun et al. [17]. (ii) Moving object segmentation based on graph cut approach, proposed by Zhou et al. [10]. (iii) Moving object segmentation based on stochastic approximation, proposed by Lopez et al. [6]. (iv) Moving object segmentation based on block-based background modelling approach, proposed by Yi et al. [8]. (v) Moving object segmentation based on spatio-temporal background modelling approach, proposed by Kim et al. [4]. (vi) Moving object detection in a video grabbed by unstable camera, proposed by Lee et al. [11]. (vii) A robust single and multiple moving object detection, tracking and classification methods proposed by Mahalingam et al. [18]. (viii) Moving object detection for a moving camera based on global motion compensation and adaptive background model, proposed by Yu et al. [19]. (ix) Moving object detection based on optical flow estimation and Gaussian mixture model, proposed by Cho et al. [20]. The performance of the proposed method was not only assessed visually but also quantitatively. For quantitative performance evaluation, we used six different evaluation measures viz. percentage of correct classification (PCC), Jaccard coefficients (JC), figure of merit (FM), quality percentage (QP), average difference (AD) and mean square error (MSE). The rest of the paper is organised as follows. Literature review is given in Section 2. An overview of different algorithms used in different stages of proposed work has been discussed in Section 3. Section 4 presents the proposed segmentation approach. Experimental results and conclusions are given in Section 5 and Section 6, respectively. 2 Literature review Several methods have been proposed for moving object segmentation in literature. In the literature, the segmentation of moving object has been mainly performed using three different classes of approaches. These approaches can be categorised as (i) Object modelling based approach [17, 21, 22]. (ii) Optical flow based approach [16, 23, 24]. (iii) Background subtraction based approach [25–28]. In object modelling based approach [17, 21, 22], target objects are extracted by applying object segmentation techniques. The methods based on this approach mainly focuses on appearance and motion coherence of moving pixels. These methods perform well when object is large and visually noticeable. This class of methods are not suitable for segmentation of tiny objects. Also, these methods are computationally costly. However, the real videos may contain objects of very small to that of large size. In optical flow-based approaches [16, 23, 24], firstly, the consecutive frames are aligned by using transformation matrices such as affine transform, homography matrix and so on, then motion flow vectors between aligned and current frame are computed. Moving foreground pixels are identified by thresholding the magnitude of motion flow vectors. The methods based on optical flow perform well in case of simple background and single object but not suitable for complex background and multiple objects. In background subtraction-based approaches [18, 25–29], moving objects are extracted by subtraction of background from the current frame. In these methods, it is assumed that pixel intensity is same over long period of time, as in static camera scenario. From past few decades, researches are working on background subtraction techniques due to their low complexity and achieved good results. However, these techniques are not suitable for varying background videos such as videos taken by PTZ and unmanned aerial vehicles cameras and so on. Therefore, considering the above facts, efficient background modelling is required for background subtraction. For background modelling, two types of approaches based on panoramic background modelling and non-panoramic background modelling have been used. In panoramic background modelling based approach [30–33], a large panorama is constructed from entire view of camera motion, by stitching input frames using image registration [34, 35], such as tonal mosaic alignment [34], Lucas–Kanade tracking [35] (LKT), scale invariant feature transform (SIFT) [36] and so on. After generating the large panorama background, moving object can be detected by applying subtraction between current frame and corresponding region in panorama background. Mittal and Huttenlocher [30] proposed background modelling method for moving camera environment and moving objects are segmented by matching Gaussian model. The idea of construction of large panorama followed by frame difference for object segmentation is exploited by Cucchiara et al. [31]. Methods proposed by Cho and Kang [32] and Xue et al. [33] have constructed panorama, using all possible view of camera, using different motion model and moving objects are segmented by existing segmentation approaches. Panoramic background methods have an advantage that can use any segmentation technique after panoramic background modelling. These methods suffer from shortcomings like registration error, parallax effect, slow initialisation and large computation time and memory requirement [4]. Shortcomings of panoramic background modelling techniques can be avoided by using non-panoramic background modelling techniques [4–11, 16, 17, 19, 20]. In this approach, background has been modelled in the size of input frames, to avoid construction of large panorama. After that, foreground has been detected by subtracting background from current frame. Normally background is in motion due to camera motion. In these approaches camera motion has been stabilised by computing a global transform matrix. The camera motion has been compensated by warping background model with current input frame. Kim et al. [4] proposed a spatio-temporal background modelling-based approach with varying learning rate for moving object segmentation. Hu et al. [5] used an integrative approach for moving object detection based on foreground feature point and foreground regions. Lopez et al. [6] has considered moving camera segmentation approach using stochastic approximation learning. Kim et al. [7] proposed a method based on clustering of optical flow vector for foreground and background pixel identification. Yi et al. [8] exploited block-based background modelling for moving object segmentation in moving background environment. Zhou et al. [10] proposed an approach for moving object segmentation in urban environment, based on graph-cut framework. Lee et al. [11] used kernel density estimation and difference of Gaussian approach for moving object detection after global camera motion stabilisation. An adaptive background modelling approach has been proposed by Yu et al. [19] in which local pixel difference and local changes between current frame and background frame has been used. Cho et al. [20] has proposed a method for moving object segmentation, in moving camera environment, which uses optical flow estimation for camera motion stabilisation and then Gaussian mixture model has been used for object segmentation. The above-mentioned methods resolve shortcomings of panoramic background approaches but non-panoramic background modelling based approaches still suffers from wrong detection of background pixels as foreground pixels (false labelling). This wrong detection is due to slow motion of object than background, abrupt change in illumination [16, 17]. Kurnianggoro et al. [16] used dense optical flow based approach for moving object segmentation, in moving camera environment based on the assumption that foreground pixels have larger magnitude and background pixels have negligible magnitude. Upon camera motion stabilisation, camera motion pixels could be brought in rest [4, 25, 26]. An adaptive background model updation is required for effective and accurate segmentation, but these approaches had not considered any background updation scheme and this is the reason why it results in poor segmentation in presence of view point changes, complex background and varying object size. Yun et al. [17] proposed a background modelling technique for moving object detection in moving camera environment based on scene changes. In this work, the problem of false labelling of background pixels as foreground pixel has been solved up to certain extent by background updation based on the scene changes, but this method is still not giving good segmentation results in presence of complex background in realistic unconstrained videos. Thus, background subtraction based methods in moving camera environment are suffering from various problems like false labelling of background pixel as moving object pixels, abrupt change in illumination and compensation error and so on. Other limitations of these methods are object occlusion, background clutter, loss of information due to 3D world projection into 2D scenes, complex and dynamic background, nearly similar intensity values of foreground and background, complex motion of object and so on. 3 Object segmentation in moving camera environment A conventional background subtraction technique fails to extract moving object in presence of camera motion as well as object motion. The work presented in this paper is based on global camera motion stabilisation for object segmentation in moving camera environment. In this section, we present an overview of different algorithms used in different stages of the proposed work in separate subsections. 3.1 Homography matrix Homography matrix is a 3 × 3 projective transform matrix of a plane, which gives point-to-point correspondence between two frames captured by moving camera [37]. To compute homography matrix between consecutive frames, at least four corresponding points are needed between them. We have used RANSAC (RANdom SAmple Consensus) algorithm [38] for computation of homography matrix. RANSAC is an iterative model for selecting inliers, among many outliers, that are compatible with homography between consecutive frames. Homography matrix is computed globally to stabilise camera motion by using extracted feature points from background frame and their corresponding points from current frame. It can be given as follows: (1)where h11, h12, h13,…h33 are nine projective points between two images of 3 × 3 projective transform matrix. The algorithm for computation of homography matrix using RANSAC is given in Fig. 1. In the algorithm, are set of initial corresponding feature points (computed using Harris corner matching) between two consecutive frames . Fig. 1Open in figure viewerPowerPoint Algorithm for computation of homography matrix 3.2 Camera motion compensation Extraction of moving object in moving camera environment is not an easy task due to the presence of background. Therefore, first of all we need to stabilise the camera motion and then by applying background subtraction technique, moving object can be extracted well. LKT is used to compute displacement of keypoints between consecutive frames. Camera motion is estimated by finding mathematical relation that maps pixel co-ordinates from one frame to the other as follows: (2)where Fi is ith feature extracted from the background frame, is ith tracked feature in current frame at the corresponding location and H is homography matrix. By multiplying homography matrix with background model, overlapped and newly covered region in the current frame are obtained, and the motion of camera is stabilised. Newly covered region are directly labelled as background area and are updated in next iteration. 3.3 Background modelling To model the background, as in fixed camera case, initial frame is considered as background frame. In this work, spatio-temporal single Gaussian background modelling technique is used to model the background [4, 25–27], i.e. single variance and single mean have been taken for each pixel. Background is updated with those pixels, which are previously labelled as background, with varying learning rate. Initially, the learning rate is taken as 1 for each background pixel and as camera moves to next frame, its value increases for overlapped region and for every newly covered region learning rate again taken as 1. Background pixel labels are decided by computing motion vectors of each pixels, as well as displacement between current pixel with neighbourhood pixel of background frame. Let xc is pixel of current frame and xb is pixel of background frame at point (x,y), then backgrounds are updated using updated mean and variances (3) (4) (5)where µt(.) and σ2t(.) are mean and variance of background at time t. L(·) is learning rate. As discussed in background modelling and updation technique, learning rate is varying and it increases with motion of camera for overlapped region. The overlapped regions are directly labelled as background pixel [4]. This labelling yields extraction of motion information of each pixel which is helpful to avoid false labelling of motionless background pixels. 3.4 Foreground and background pixel identification After camera motion stabilisation, foreground pixels are identified by subtracting the background frame from the overlapped region of current frame. The conventional approach of segmentation of moving object suffers from problems like registration error, parallax effect, slow initialisation, huge memory uses, abrupt change in illumination and false labelling of moving pixels by background pixel. Here in the proposed work, the shortcomings like registration error, slow initialisation, huge memory uses and parallax effect have been resolved by modelling background in the size of current frame. Since background and foreground both have different velocities, therefore the problem of abrupt change in illumination and false detection of moving pixels as background pixels have been resolved by computing motion flow vector of each pixel. Moving pixels are identified by computing labels for deciding background and foreground pixels by comparing current frame with background model followed by thresholding the motion vector of each pixel as follows. The label for the decision of foreground\background is calculated as (6)Magnitude of motion vectors for each pixel is computed as (7)where and are optical flow vectors in x and y directions. Foreground and background pixels are labelled as (8) (9)where and are mean and variance of background model at time t, and are thresholds used for identification of background and foreground pixels, respectively, and is intensity value of the current frame. Foreground and background pixels are identified based on labels of pixels followed by thresholding of motion flow vectors which are computed using dense optical flow approach. Motion flow vectors help in avoiding the false identification of motionless pixel of background. This is due to negligible magnitude of motion vector of background pixels compared to moving pixels of foreground. However, the segmentation results may be affected by the presence of noise and nearly same intensity levels of neighbouring pixels of object. To overcome this problem results could be further refined by post-processing morphological operation. In this work, we used dilation and erosion morphological operators to refine the result with diagonal and ellipse structuring elements of size 3 × 3 and 5 × 5. 3.5 Dense optical flow Optical flow computes the vector fields that determine velocity of each moving pixel in time-varying image sequences [12, 15]. Dense optical flow captures the motion of each pixel in consecutive frames, i.e. it shows displacement of intensity pattern between consecutive frames in the form of point-to-point correspondence [12–15, 39, 40] of pixels. There are several approaches for motion estimation through optical flow such as correlation or block matching [15], feature tracking [39], energy-based approaches [39], pixel intensity-based approaches (brightness remains consistent with time) [13], gradient-based approaches [12] and polynomial expansion transform-based approaches [40]. In this work, we use polynomial expansion transform-based approach proposed by Gunnar Farneback [40]. The first step of this approach is to approximate the neighbourhood of the consecutive frames using polynomial expansion transform. Then displacement vectors are estimated using these polynomial expansions after number of refinements [40]. 4 Proposed method Objective of this work is to present a fast and reliable method for object segmentation in moving camera environment for realistic and unconstrained videos. Object segmentation in moving camera environment is not easy tasks due to the presence of two types of motion – background motion and object motion. Therefore, first task is to stabilise camera motion, so that the frame sequences could be treated as in the case of static camera. After stabilisation of camera motion, single Gaussian background modelling approach is used for background modelling that results in improved segmentation at low computational cost. Hybridisation of existing background subtraction technique with camera motion compensation still suffers from slow initialisation, abrupt change in illumination, appearance changes, compensation error and false labelling problem [16, 17], as discussed in Sections 2 and 3. To handle these problems, a new approach using dense optical flow has been proposed for differentiating between foreground and background pixels. Computation of labels corresponding to each pixel as foreground or background pixel, if we revalidate each pixel as a foreground or background pixel through motion vector of each pixel, we can resolve the problem of illumination change and false labelling. This is due to a larger magnitude of the object pixels compared to the background pixels in motion flow vector. Therefore, use of dense optical flow, the motion flow vector of each pixel has been computed for the correct identification of moving object pixels. The block diagram of the proposed method is shown in Fig. 2. Step-wise illustration of Fig. 2 is as follows: (i) Initialise the first frame as background frame and update it in each iteration with new frame after camera motion estimation, using single Gaussian background modelling technique. Fig. 2Open in figure viewerPowerPoint Block diagram of the proposed approach (ii) Extract the feature vector from the background frame. Harris corner feature [41] has been used as feature due to its invariance nature w.r.t. rotation and translation, and varying illumination condition [42]. (iii) Track the corresponding extracted feature in the current frame using LKT [34] to register two consecutive frames. (iv) Compute homography matrix [37] for computing point correspondences in consecutive frames. (v) Stabilise motion of camera by computing overlapped region and newly covered region by warping background with current frame. (vi) To identify object pixels among foreground and background pixels, compute motion flow vector of each pixel between background and current frame. Foreground and background pixels are differentiated by subtracting background from current frame on the basis of computed labels [26] followed by thresholding the magnitude of motion flow vector of each pixel. (vii) Results obtained are further processed by morphological operation to refine the extracted moving objects (see Fig. 3). Fig. 3Open in figure viewerPowerPoint Algorithm for proposed moving object segmentation 5 Experimental results and discussion The proposed method of moving object segmentation in moving camera scenario is implemented in C + + environment with computer vision open source library 3.0. We performed experiments on number of realistic unconstrained videos recorded in various real scenarios like cloudy, fog and rainy weather. We tested our method on five self-created datasets (https://drive.google.com/open?id=1o_37sfIDOaO3HW99TwMYVWLT_tT3CeJ7) and one standard dataset used by Yun et al. [17] (http://pil.snu.ac.kr/user/kmyun/file/scbudata.zip). The dataset [17] consists of various movie clips of dancing, skating, swing, walking and so on. For representative purpose, in the paper, results on three datasets are presented. However, all the results have been uploaded on the website (https://drive.google.com/file/d/1LhTz6pWanA67byapu2b5EJWbpYB1FJEu). The proposed method is compared with other state-of-art methods proposed by Yun et al. [17], Zhou et al. [10], Lopez et al. [6], Yi et al. [8], Kim et al. [4], Lee et al. [11], Mahalingam et al. [18], Yu et al. [19] and Cho et al. [20] for all representative video sequences. The segmentation results on some representative frames by the proposed method and other methods used in comparison have been shown in Figs. 4, 5–6. A complete set of results on different videos are available at https://drive.google.com/file/d/1LhTz6pWanA67byapu2b5EJWbpYB1FJEu. The qualitative results are not sufficient to judge the performance of the segmentation. Therefore, we have also evaluated the performance quantitatively. For quantitative evaluation, we have constructed ground truth by publicly available tool Virtual Dub (http://www.virtualdub.org/index.html). In this paper, we used six standard evaluation metrics – PCC, JC, FM, QP, AD and MSE [9, 29, 43]. The metric values on different datasets have been given in Tables 1–3. Fig. 4Open in figure viewerPowerPoint Segmentation results on a video clip of frame size 640 × 360, frames are arranged from top to bottom as (a) Frame no. 50, (b) Frame no. 450, (c) Frame no.725, (d) Frame no. 925 Fig. 5Open in figure viewerPowerPoint Segmentation results on a video clip of frame size 640 × 272, frames are arranged from top to bottom as (a) Frame no. 120, (b) Frame no. 450, (c) Frame no. 950, (d) Frame no. 1060 Fig. 6Open in figure viewerPowerPoint Segmentation results on a video clip of frame size 320 × 240. Frames are arranged from top to bottom as (a) Frame no. 50, (b) Frame no. 400, (c) Frame no. 700, (d) Frame no. 900 Table 1. Performance measures for the proposed and other methods for representative segmented frames Frame no. Method PCC JC FM QP AD MSE 50 Yun et al. [17] 0.046 0.024 0.023 2.380 0.135 0.844 Zhou et al. [10] 0.037 0.019 0.018 1.910 0.167 0.858 Lopez et al. [6] 0.237 0.134 0.118 13.45 01.69 0.975 Yi et al. [8] 0.141 0.076 0.070 7.590 0.328 0.945 Kim et al. [4] 0.379 0.234 0.189 23.430 0.423 0.978 Lee et al. [11] 0.048 0.024 0.024 2.460 0.120 0.278 Mahalingam et al. [18] 0.312 0.185 0.156 18.520 0.064 0.253 Yu et al. [19] 0.045 0.023 0.022 2.340 0.007 0.379 Cho et al. [20] 0.080 0.042 0.040 4.200 0.134 0.213 proposed 0.430 0.274 0.215 27.430 0.106 0.115 450