An image enhancement algorithm of video surveillance scene based on deep learning

IET Image ProcessingVolume 16, Issue 3 p. 681-690 ORIGINAL RESEARCH PAPEROpen Access An image enhancement algorithm of video surveillance scene based on deep learning Wei-wei Shen, Wei-wei Shen Institute of Information Engineering, Suqian University, Suqian, ChinaSearch for more papers by this authorLin Chen, Lin Chen Institute of Information Engineering, Suqian University, Suqian, ChinaSearch for more papers by this authorShuai Liu, Corresponding Author Shuai Liu liushuai@hunnu.edu.cn orcid.org/0000-0001-9909-0664 Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha, China Hunan Xiangjiang Artificial Intelligence Academy, Hunan Normal University, Changsha, China Correspondence Shuai Liu, Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha 410081, China. Email: liushuai@hunnu.edu.cn Yu-Dong Zhang, School of Informatics, University of Leicester, Leicester LE1 7RH, UK. Email: yudongzhang@ieee.orgSearch for more papers by this authorYu-Dong Zhang, Corresponding Author Yu-Dong Zhang yudongzhang@ieee.org School of Informatics, University of Leicester, Leicester, UK Correspondence Shuai Liu, Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha 410081, China. Email: liushuai@hunnu.edu.cn Yu-Dong Zhang, School of Informatics, University of Leicester, Leicester LE1 7RH, UK. Email: yudongzhang@ieee.orgSearch for more papers by this author Wei-wei Shen, Wei-wei Shen Institute of Information Engineering, Suqian University, Suqian, ChinaSearch for more papers by this authorLin Chen, Lin Chen Institute of Information Engineering, Suqian University, Suqian, ChinaSearch for more papers by this authorShuai Liu, Corresponding Author Shuai Liu liushuai@hunnu.edu.cn orcid.org/0000-0001-9909-0664 Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha, China Hunan Xiangjiang Artificial Intelligence Academy, Hunan Normal University, Changsha, China Correspondence Shuai Liu, Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha 410081, China. Email: liushuai@hunnu.edu.cn Yu-Dong Zhang, School of Informatics, University of Leicester, Leicester LE1 7RH, UK. Email: yudongzhang@ieee.orgSearch for more papers by this authorYu-Dong Zhang, Corresponding Author Yu-Dong Zhang yudongzhang@ieee.org School of Informatics, University of Leicester, Leicester, UK Correspondence Shuai Liu, Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha 410081, China. Email: liushuai@hunnu.edu.cn Yu-Dong Zhang, School of Informatics, University of Leicester, Leicester LE1 7RH, UK. Email: yudongzhang@ieee.orgSearch for more papers by this author First published: 24 June 2021 https://doi.org/10.1049/ipr2.12286AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Target enhancement is the most important task in a video surveillance system. In order to improve the accuracy and efficiency of target enhancement, and better deal with the subsequent recognition, tracking, behaviour understanding and other processing of targets, a deep learning-based image enhancement algorithm for video surveillance scenes is proposed. First, the super-resolution reconstruction of the image is carried out through the image super-resolution reconstruction method based on the hybrid deep convolutional network to improve the sharpness of the image. Then, for the reconstructed video surveillance scene image, the watershed image enhancement algorithm based on morphology and region merging is used to realize the enhancement of the video surveillance scene image. Deep learning algorithms can improve the accuracy of image enhancement through iterative calculations. Experimental results show that after image enhancement in daytime, night and noisy video surveillance scenes, the maximum enhancement difference rate is less than 0.5%, the cross-linking degree is close to 1, and the average image enhancement time is less than 1.3 s. It can realize image enhancement of video surveillance scenes and improve the image clarity of the video surveillance scene. 1 INTRODUCTION In daily life, human beings get most information by using visual system, which belongs to image information, and so image is an irreplaceable information carrier for human perception of the world [1]. With the development of science and technology, humans use computer to simulate the process of human brain recognizing the target in the image, in order to get the method of image processing, and make the image more suitable for human eye observation or convenient for instrument detection [2]. In the field of computer vision application research, image enhancement is the first step, which is very important in the process of digital image processing. If the quality of image enhancement results is very good, it will be of great help to the next step of target recognition, image recognition, scene analysis and other work. Therefore, the research of image enhancement is of great significance for image processing, image analysis and other fields. The research of image enhancement algorithm has made great progress, but because of its complexity, there are still many problems to be solved. So far, there is no image enhancement method that can be applied to all fields, and there is no general evaluation standard [3]. Traditional image enhancement methods are difficult to extract features, and the enhancement effect is difficult to fully meet various requirements, so there is still a lot of room for improvement. People are still trying to combine the application of advanced algorithms into the field of image enhancement, hoping to produce better enhancement results. Therefore, further research on image enhancement algorithm is still of great practical significance [4]. In the research and application of image, people are often interested in some parts of the image, which are often called targets or prospects. They generally correspond to specific regions with unique properties in the image. In order to identify and analyse the objects in the image, it is necessary to separate them from the image [5]. On this basis, people can further measure the target and use the image. With the development of computer technology and digital image monitoring technology, the first level targets of military key alert have begun to use the digital image monitoring system to improve the monitoring efficiency [6]. In order to further improve the intelligent degree of the monitoring system, it is necessary to analyse the image effectively. Video surveillance scene image enhancement is an important image analysis technology [7]. Related scholars have done research on image enhancement algorithms for video surveillance scenes, such as signal-to-noise ratio (PSNR) [8] and structural similarity (SSIM) [9]. The main advantage of PSNR algorithm is that the algorithm does not need parameter control, and the enhancement accuracy is relatively high, but the algorithm needs to calculate the similarity between adjacent regions. As the most basic calculation unit, the SSIM is an index that measures the similarity of two images. It can improve the accuracy of image enhancement, but the algorithm takes a long time, resulting in low efficiency of the algorithm. Deep learning is a branch of machine learning, which is a new direction in the recent decade. The essence of deep learning is bionics, which simulates the process of human brain perception and cognition. It is the research goal of deep learning to construct the neural network of human brain for analytical learning and imitate the mechanism of visual system to analyse image, sound and text information. Deep learning is developed on the basis of artificial neural network. At the end of the last century, neural network was once paid great attention to in the field of machine learning, but later, there appeared some problems, such as easy fitting, difficult to adjust parameters, slow training speed, and not more efficient than other methods when there were fewer levels, so it gradually faded out [10]. Until 2006, Geoffrey Hinton put forward the idea of deep learning. Subsequently, deep learning has been paid attention to by many scholars in academia and widely used in industry. It has made a breakthrough in the fields of speech recognition, image recognition and natural language processing. The difference between deep learning and traditional shallow learning is that deep learning requires deep network structure and attaches great importance to feature learning. Compared with the method of artificial feature construction, it is easier to obtain all kinds of connotative information by using massive data for training and learning features. At present, deep learning has made remarkable achievements in many fields, and has become a research hotspot that attracts the attention of scholars in the industry. Moreover, the upsurge of deep learning is still continuing, and continuous major breakthroughs have been made, which will have a significant impact on the field of machine learning. In this paper, deep learning technology is used in the video surveillance scene image enhancement, and an image enhancement algorithm of video surveillance scene based on deep learning is proposed to realize the video surveillance scene image enhancement. 2 IMAGE ENHANCEMENT ALGORITHM OF VIDEO SURVEILLANCE SCENE BASED ON DEEP LEARNING This chapter is based on the super-resolution (SR) reconstruction of the image based on the hybrid deep convolutional network, which is based on the image background and target characterized by morphological opening, closing mixing operation and the Laplacian sharpening technology. The contour is then enhanced by the Laplacian operator. According to the brightness of the video surveillance scene image area, the object and background area of the video surveillance scene image are marked, and the modified gradient amplitude image watershed is segmented. Finally, RMMS is used to merge the segmented image regions, and to merge the missing labelled regions into the target or background regions until there is no new region merging algorithm. The above steps can effectively improve the clarity of the reconstructed image and improve the accuracy of the reconstruction, and are used to implement an image enhancement algorithm for a video surveillance scene. 2.1 Image SR reconstruction method based on hybrid deep convolution network Super resolution (SR) reconstruction aims to reconstruct high-resolution (HR) images with rich details by inputting one or more low-resolution (LR) images. As an ill posed inverse problem, reconstruction technology needs to collect and analyse as many adjacent pixels as possible to obtain more clues, which can be used to supplement the missing pixel information in the process of up-sampling. The signal image super resolution (SISR) reconstruction of video surveillance scene image is to use the rich information contained in a video surveillance scene image and the visual priors obtained from the sample image to identify important visual clues, fill in details, and present as faithfully and aesthetically as possible. In recent years, reconstruction technology has been widely used in medical image, satellite remote sensing, video surveillance and other fields [11]. This paper proposes an image SR reconstruction method based on hybrid deep convolution network, which is one of the core technologies in the field of deep learning. In this method, the codec structure constructed by convolution and deconvolution can remove the noise generated in the process of image reconstruction. Through the mixed use of different convolution methods, the end-to-end network is constructed, and the SR image in line with the original image is reconstructed [12]. In this paper, deep learning technology is used in the video surveillance scene image enhancement, and an image enhancement algorithm of video surveillance scene based on deep learning is proposed to realize the video surveillance scene image enhancement. Through the image SR reconstruction method based on the hybrid deep convolutional network, the image is SR reconstruction to improve the sharpness of the image. Then, for the reconstructed video surveillance scene image, the watershed image enhancement algorithm based on morphology and region merging is used to realize the enhancement of the video surveillance scene image. 2.1.1 Up-sampling In the task of SR reconstruction of video surveillance scene image based on deep learning, the reconstruction model of high-resolution image (formula 1) is used. In the up-sampling module, the LR image of video surveillance scene is scaled to a sub LR image with the same number of pixels as the high-resolution image [13]. The simplest way of up-sampling is resampling and interpolation: the input image is rescaled to a desired size, and the pixels of each point are calculated at the same time. In this paper, a deconvolution network structure is used to sample the input LR image to the target image size [14]. In this part, a deconvolution layer with convolution kernel of 3 × 3 is used to sample the LR image of the video surveillance scene to the same scale of the target image as the input of the feature extraction layer. I H = D − 1 I L B − 1 − S \begin{equation}{I^H} = {D^{ - 1}}{I^L}{B^{ - 1}} - S\end{equation} (1)where I H ${I^H}$ is the high-resolution image of video surveillance scene; D − 1 ${D^{ - 1}}$ is the up-sampling operation; I L ${I^L}$ is the low-resolution image of video surveillance scene; B − 1 ${B^{ - 1}}$ is the deblurring operation; − S $ - S$ is the noise reduction operation. 2.1.2 Image feature extraction In traditional image restoration, a strategy of image feature extraction is to extract image blocks intensively, and then use a set of pre trained bases (such as principal component analysis (PCA), discrete cosine transform (DCT) etc.) to represent them. In convolution neural network, this part can be incorporated into the basic optimization of the network, and convolution operation can automatically extract video surveillance scene image features [15]. Formally, the sub network of feature extraction in this paper is represented as follows: F 1 X = max 0 , W 1 ⊗ Z + b 1 \begin{equation}{F_1}\left( X \right) = \max \left( {0,{W_1} \otimes Z + {b_1}} \right)\end{equation} (2)where Z $Z$ is the LR image of the video surveillance scene after up-sampling; W 1 ${W_1}$ and b 1 ${b_1}$ represent the convolution weight and offset, respectively, where the size of W 1 ${W_1}$ is 3 × 3 × 64 $3 \times 3 \times 64$ ; ⊗ $ \otimes $ is the convolution operation, 0 boundary is added and step size is 1, so that the input and output sizes are consistent and the boundary rank reduction is prevented. 2.1.3 Structure of codec denoising In the feature denoising structure, convolution and deconvolution are cascaded to construct the encoding and decoding structure, which can eliminate the feature noise of video surveillance scene image to the maximum extent. Convolution layer retains the main video surveillance scene image content, while deconvolution layer is used to compensate for the details, so as to achieve good denoising effect and better retain the video surveillance scene image content. Formally, this layer is expressed as: H 1 F 1 = max 0 , W 2 ⊗ F 1 + b 2 \begin{equation}{H_1}\left( {{F_1}} \right) = \max \left( {0,{W_2} \otimes {F_1} + {b_2}} \right)\end{equation} (3) H 2 F 1 = max 0 , W 3 ⊗ H 1 + b 3 \begin{equation}{H_2}\left( {{F_1}} \right) = \max \left( {0,{W_3} \otimes {H_1} + {b_3}} \right)\end{equation} (4) where the output in the image feature extraction stage of video surveillance scene is F 1 ${F_1}$ ; W 2 ${W_2}$ and b 2 ${b_2}$ are convolution weight and offset size. W 3 ${W_3}$ and b 3 ${b_3}$ are deconvolution weights and offsets; H 1 ( F 1 ) ${H_1}( {{F_1}} )$ and H 2 ( F 1 ) ${H_2}( {{F_1}} )$ are the image features of video surveillance scene extracted by deconvolution one operation and two operations in turn; H 1 ${H_1}$ is deconvolution one operation. The convolution layer gradually reduces the size of the feature map, retains the main image information of the video surveillance scene, and obtains the abstract content of the image features of the video surveillance scene. The deconvolution layer gradually increases the size of the feature map, enlarges the size of the feature map, and restores the detail information of the image features of the video surveillance scene. At the same time, jump connection is used to speed up the training process, so as to ensure that the input and output sizes of the encoding and decoding structure are consistent. At the same time, it also ensures the test efficiency in the case of limited computing power of the mobile terminal, and obtains the image feature map of the video surveillance scene to remove the noise [16]. All 64 convolution cores of 3 × 3 × 64 are used in this part of the convolution layer. However, the first half of the convolution operation does not add 0 boundary and the step size is 2, so that the output size of the feature becomes half of the input size. In the second part of deconvolution feature recovery, 0 boundary is added and the step size is 2. The video surveillance scene feature map is restored to the original size to ensure the image scale integrity. 2.1.4 Reconstruction of video surveillance scene image In the process of video surveillance scene image reconstruction, the input of hidden state feature image F 2 ${F_2}$ and the output of SR reconstruction image can be regarded as the inverse operation of feature extraction stage. In network convolution, convolution kernel W c d $W_c^d$ is used as a reaction base coefficient, and each position of high-dimensional hidden state image feature is regarded as SR image domain to obtain SR reconstruction image. For this reason, a convolution layer is defined to generate the final SR image I S R ${I^{SR}}$ I S R = W c d ⊗ F 4 + b 5 \begin{equation}{I^{SR}} = W_c^d \otimes {F_4} + {b_5}\end{equation} (5)where W c d $W_c^d$ is c convolutions with the size of 1 × 1 × 64, c is the number of video surveillance scene image channels, b 5 ${b_5}$ is the convolution bias, and F 4 ${F_4}$ is the convolution kernel after four convolutions. The first four convolutions in this part use 64 convolutions with 3 × 3 × 64 to check the image features of video surveillance scene for high-dimensional extraction. The convolution kernel convolution point spacing is 1, 1, 2 and 4, respectively, which can calculate the image features with a larger receptive field. Then, a convolution layer composed of 64 convolution cores with 3 × 3 × 64 is used to sum the first extracted image features of video surveillance scene with the high-dimensional features to ensure the full utilization of the features. Finally, the required SR image is reconstructed by using the convolution layer of c convolution cores with 1 × 1 × 64. 2.1.5 Loss function and optimization For any given training data set { X C 1 , Y C 1 } i = 0 n − 1 $\{ {X_C^1,Y_C^1} \}_{i = 0}^{n - 1}$ , the goal is to find an accurate mapping value F ( x ) $F( x )$ , so as to minimize the mean square error (MSE) between the SR reconstructed image I S R ${I^{SR}}$ and the real image T $T$ , and improve the image quality evaluation index—peak signal-to-noise ratio (PSNR). Although high PSNR does not represent absolute excellence of reconstructed image, satisfactory performance is still observed when alternative evaluation indexes are used to evaluate the model. M E S = 1 H ∗ W ∑ i = 0 H − 1 ∑ j = 0 W − 1 P i , j − T i , j 2 \begin{equation}MES = \frac{1}{{H*W}}\sum_{i = 0}^{H - 1} {\sum_{j = 0}^{W - 1} {{{\left( {P\left( {i,j} \right) - T\left( {i,j} \right)} \right)}^2}} } \end{equation} (6)where P ( i , j ) $P( {i,j} )$ and T ( i , j ) $T( {i,j} )$ represent the predicted image and the real image, respectively; i and j are the reconstruction times of the height and width of the video surveillance scene image; H and W are the height and width of the video surveillance scene image, respectively. f l o s s = M S E + β θ 2 \begin{equation}{f_{loss}} = MSE + \beta {\left\| \theta \right\|^2}\end{equation} (7) That is, f l o s s = 1 N ∑ i = 0 N − 1 F x − Y 2 + β θ 2 \begin{equation}{f_{loss}} = \frac{1}{N}\sum_{i = 0}^{N - 1} {{{\left\| {F\left( x \right) - Y} \right\|}^2} + \beta {{\left\| \theta \right\|}^2}} \end{equation} (8)where f l o s s ${f_{loss}}$ is the loss function; β ∥ θ ∥ 2 $\beta {\| \theta \|^2}$ is the multiplicative coefficient of weight attenuation; N is the optimization times. 2.2 Watershed image enhancement algorithm based on morphology and region merging 2.2.1 Morphological open–close hybrid operation Mathematical morphology is a tool for mathematical analysis of images based on structural elements. In the process of mathematical analysis, structural elements with certain morphological structure are used to measure and extract the corresponding shape in the image, so as to further achieve the purpose of image recognition and analysis. In the process of image processing, there are four basic operations of mathematical morphology: expansion, corrosion, open operation and closed operation. Image etching can remove small protrusions or burr of objects smaller than structural elements. By selecting structural elements of different sizes, objects of different sizes can be removed from the image. The expansion operation can enlarge the image and fill the holes in the object. In morphology, structural element is the most basic concept. The function of structural elements in morphological transformation is equivalent to 'filter window' in signal processing. B ∈ B ( x ) $B \in B( x )$ is used to represent the target structure element in the video surveillance scene image. For each point x in the video surveillance scene image space E, the operations of corrosion X and expansion Y are defined as follows: X = E ∘ B = X : B x ⊂ E \begin{equation}X = E \circ B = \left\{ {X:B\left( x \right) \subset E} \right\}\end{equation} (9) Y = E ⊕ B = Y : B y ∩ E ≠ ∅ \begin{equation}Y = E \oplus B = \left\{ {Y:B\left( y \right) \cap E \ne \emptyset } \right\}\end{equation} (10) Morphological open–close hybrid operation can achieve the goal of eliminating the hole of target details in the image on the premise of maintaining the integrity and position of target information. This step can eliminate the local extreme caused by irregular grey disturbance and noise in the gradient image before the subsequent processing, and preserve the object contour and information of the video surveillance scene image. 2.2.2 Laplacian sharpening The edge and details of the video surveillance scene image are further emphasized, and the video surveillance scene image is sharpened to improve the contrast [17]. Laplacian sharpening image is related to the mutation degree of the surrounding pixels of a pixel to this pixel, and the transition degree of the pixel can be found according to the second differential. When the grey value of the centre pixel in the neighbourhood is lower than the average grey value of the neighbourhood, the grey value of the centre pixel should also decrease; when the grey value of the centre pixel is higher than the average grey value of the neighbourhood, the grey value of the centre pixel should increase. In this way, image sharpening of video surveillance scene is realized [18]. The relationship between second-order differential and pixel is as follows: ∂ f ∂ x = f x , y − f x − 1 , y ∂ f ∂ y = f x , y − f x , y − 1 \begin{equation}\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {\frac{{\partial f}}{{\partial x}} = f\left( {x,y} \right) - f\left( {x - 1,y} \right)}\\[6pt] {\frac{{\partial f}}{{\partial y}} = f\left( {x,y} \right) - f\left( {x,y - 1} \right)} \end{array} } \right.\end{equation} (11) where the input video surveillance scene image is f ( x , y ) $f( {x,y} )$ , x and y represent the pixels of the image. The central mask coefficient of Laplace is as follows: ∇ 2 f = 4 f x , y − f x − 1 , y − f x + 1 , y − f x , y − 1 \begin{equation}{\nabla ^2}f = 4f\left( {x,y} \right) - f\left( {x - 1,y} \right) - f\left( {x + 1,y} \right) - f\left( {x,y - 1} \right)\end{equation} (12) Four neighbourhood matrix templates are obtained. 0 − 1 0 − 1 4 − 1 0 − 1 0 \begin{equation}\left( { \def\eqcellsep{&}\begin{array}{@{}*{3}{c}@{}} 0&\quad { - 1}&\quad 0\\ { - 1}&\quad 4&\quad { - 1}\\ 0&\quad { - 1}&\quad 0 \end{array} } \right)\end{equation} (13) Similarly, the sharpening process of eight neighbourhood of video surveillance scene image is as follows: ∇ 2 f = 8 f x , y − f x − 1 , y − 1 − f x − 1 , y − f x − 1 , y + 1 − f x , y − 1 − f x , y + 1 − f x + 1 , y − 1 − f x + 1 , y − f x + 1 , y + 1 \begin{equation} \def\eqcellsep{&}\begin{array}{l} {\nabla ^2}f = 8f\left( {x,y} \right) - f\left( {x - 1,y - 1} \right) - f\left( {x - 1,y} \right)\\ -\, f\left( {x - 1,y + 1} \right) - f\left( {x,y - 1} \right) - f\left( {x,y + 1} \right) \\ -\, f\left( {x + 1,y - 1} \right) - f\left( {x + 1,y} \right) - f\left( {x + 1,y + 1} \right) \end{array} \end{equation} (14) The matrix template is: − 1 − 1 − 1 − 1 8 − 1 − 1 − 1 − 1 \begin{equation}\left( { \def\eqcellsep{&}\begin{array}{@{}*{3}{c}@{}} { - 1}&\quad { - 1}&\quad { - 1}\\ { - 1}&\quad 8&\quad { - 1}\\ { - 1}&\quad { - 1}&\quad { - 1} \end{array} } \right)\end{equation} (15) By replacing the value at the original pixel (x,y) with the obtained value, similar boundary can be obtained, and the sharpened video surveillance scene image can be obtained by the following formula: g x , y = f x , y − ∇ 2 f x , y , ∇ 2 f x , y < 0 f x , y + ∇ 2 f x , y , ∇ 2 f x , y ≥ 0 \begin{equation}g\left( {x,y} \right) = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {f\left( {x,y} \right) - {\nabla ^2}f\left( {x,y} \right),{\nabla ^2}f\left( {x,y} \right) < 0}\\[4pt] {f\left( {x,y} \right) + {\nabla ^2}f\left( {x,y} \right),{\nabla ^2}f\left( {x,y} \right) \ge 0} \end{array} } \right.\end{equation} (16) 2.2.3 Marking background and target When looking for the mark, the target object is relatively bright, while the background corresponds to the dark area. If the background area of the video surveillance scene image is dark at this time, the image binarization can be used to find the corresponding area of the background, so as to ensure that the external markers can be included in the background. Because the target object in the video surveillance scene image is generally located in the prominent position of other objects in the image, the target object may be the largest area of the local area pixel value, which can be found by finding the local area extreme value. The large value is used as the marker of the target object [19]. The process of finding internal and external markers is shown in Figure 1. FIGURE 1Open in figure viewerPowerPoint The process of finding inner and outer markers First, the video surveillance scene image is continuously opened and closed to generate the maximum and minimum value of the image area, and the maximum value point is marked as the target. Then, the video surveillance scene image is binarized, and the pixel position information is transformed into the grey information of the video surveillance scene image by distance transformation. Then, the boundary between the adjacent areas of the video surveillance scene image is taken out to form the background marker, and the gradient amplitude image of the video surveillance scene image is modified. 2.2.4 Region merging based on maximum similarity (RMMS) After the marking in the previous step, there are still some missing unmarked areas in the video surveillance scene image. This method can transform the merged region to the target region or background region. RMMS is divided into two stages. The first stage is sampling iteration method, which merges the unmarked regions which are more similar to the background set until there is no new region in the background set to merge with a certain region of the unmarked set. In each iteration step, the background se