Two‐stage image smoothing based on edge‐patch histogram equalisation and patch decomposition

IET Image ProcessingVolume 14, Issue 6 p. 1132-1140 Research Article Free Access Two-stage image smoothing based on edge-patch histogram equalisation and patch decomposition Yepeng Liu, orcid.org/0000-0001-6340-7818 School of Software, Shandong University, Jinan, 250101 People's Republic of ChinaSearch for more papers by this authorXiang Ma, School of Software, Shandong University, Jinan, 250101 People's Republic of ChinaSearch for more papers by this authorXuemei Li, Corresponding Author xmli@sdu.edu.cn School of Software, Shandong University, Jinan, 250101 People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, Yantai, 264025 People's Republic of ChinaSearch for more papers by this authorCaiming Zhang, School of Software, Shandong University, Jinan, 250101 People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, Yantai, 264025 People's Republic of China Digital Media Technology Key Lab of Shandong Province, Jinan, 250014 People's Republic of ChinaSearch for more papers by this author Yepeng Liu, orcid.org/0000-0001-6340-7818 School of Software, Shandong University, Jinan, 250101 People's Republic of ChinaSearch for more papers by this authorXiang Ma, School of Software, Shandong University, Jinan, 250101 People's Republic of ChinaSearch for more papers by this authorXuemei Li, Corresponding Author xmli@sdu.edu.cn School of Software, Shandong University, Jinan, 250101 People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, Yantai, 264025 People's Republic of ChinaSearch for more papers by this authorCaiming Zhang, School of Software, Shandong University, Jinan, 250101 People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, Yantai, 264025 People's Republic of China Digital Media Technology Key Lab of Shandong Province, Jinan, 250014 People's Republic of ChinaSearch for more papers by this author First published: 06 April 2020 https://doi.org/10.1049/iet-ipr.2019.0484Citations: 3 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Part of important structural edges in the image is smoothed due to the small gradients, while the others are preserved with greater gradients. Therefore, the authors propose a two-stage image smoothing method based on edge-patch histogram equalisation and patch decomposition. The authors' purpose is to increase the gradient of important structural edges while reducing the gradient of the texture region. Therefore, they divide the image into edge-patches where the structural edges are concentrated or non-edge-patches where the texture details are concentrated by image segmentation. The edge-patch needs to be equalised by the histograms for increasing the gradient of the edge pixels. All patches are decomposed to extract the smooth component for reducing the gradient of pixels. The smooth component of each patch is smoothed via gradient minimisation. In order to ensure the continuity of the patch boundaries, the edge-patch is inversely equalised. Finally, the whole image is smoothed via gradient minimisation for removing residual textures and seams. Experimental results demonstrate that the proposed method is more competitive in maintaining important structural edges and removing texture details than the state-of-the-art approaches. The proposed method can be applied to many areas of image processing. 1 Introduction Image smoothing is an important research topic in image processing. It has many applications, such as edge detection, image enhancement, image abstraction, content-aware image resizing and so on. In general, natural images contain both clear structural edges of objects and rich details. Due to the complexity of natural images, it is difficult to give a clear definition of structural edges and texture details, which is a major problem with image smoothing. Psychological studies show that the human visual system can easily understand natural images because human is sensitive to the main structural edges and does not pay attention to texture details [1]. Image smoothing is to achieve the same purpose, preserving the main structural edges of the objects and removing texture details. In recent decades, image smoothing has attracted the attention of many outstanding scholars, and there have been many algorithms for image smoothing. According to the number of times the image is smoothed, they are divided into non-iterative and iterative smoothing algorithms. Non-iterative algorithms get the smooth results by smoothing the images once. They can be divided into edge-preserving and structure-preserving smoothing algorithms. Edge-preserving smoothing algorithm is to remove trivial details while preserving major image edges (edge-preserving). Most of them use smoothing operators to distinguish details from major structures. The bilateral filter [2] is one of the representative operators, which is proposed by Tomasi and Manduchi in 1998. The method is non-iterative, local, and simple, it may have unwanted gradient reversal artefacts near edges. In 2008, Farbman et al. [3] propose weighted least square (WLS) filter. It advocates the use of an alternative edge-preserving smoothing operator, which is particularly well suited for progressive coarsening of images and for multi-scale detail extraction. While WLS can get high quality images, it comes with the price of expensive computational time. In 2009, Subr et al. [4] define details as oscillations between local minimum and maximum. The details are captured by capturing local spatial oscillations by multi-scale image decomposition. While the texture details are smoothed, the structural edges become blurred. In 2011, Paris et al. [5] propose local Laplacian filter. It characterises edges with a simple threshold on pixel values that can differentiate large-scale edges from small-scale details. The method is simple and flexible. In 2012, He et al. [6] propose the guided image filtering. It can be used as an edge-preserving smoothing operator like BLF, but it works better in detail preservation. However, there are halos near some edges. In 2016, Deng proposes a new guided wavelet shrinkage (GWS) smoothing method. It can also be configured to operate in an iterative or non-iterative way. There are artefacts around sharp edges when the level of noise is high. And the textures near sharp edges are difficult to be removed. Structure-preserving smoothing algorithms are based on the following assumptions: sliding windows containing different patterns (structure and details) behave differently with suitable measurement. Therefore, structure-preserving algorithms use the statistical features within local sliding window to remove details and extract structures. In 2013, Bao et al. [7] propose the tree filter. A tree distance and the two factors in BLF are used for weighted-averaging. However, there are residual texture details or some structural edges disappear in the method. In 2013, Karacan et al. [8] propose an approach which depends on region covariance matrices of simple image features. The second-order statistics as a patch descriptor is used to effectively implicitly capture local structure and texture information. Some image structures may be misinterpreted as textures when they are statistically similar structures. In 2014, Zhang et al. [9] use a space-filling curve to perform the separation of edges and details. The smoothing effect is achieved by modulating local extrema after empirical mode decomposition. It is implemented on a one-dimensional curve instead of a two-dimensional image grid. Salient edges become blurred after multiple iterations. In 2011, Kyprianidis and Kang. [10] propose a non-realistic rendering technique to create stylised abstractions from colour images and videos, which is fast and mainly used for video processing. However, it is not good for information-rich images. Iterative algorithm needs repeated calculation to get smooth image. In 1992, Rudin et al. [11] propose the total variation (TV) regular term with norm of gradient map. Then the total variation is rapidly expanded and applied in image processing. In 2011, Xu et al. [12] present an image editing method via gradient minimisation, which is effective for sharpening major edges by increasing the steepness of transition while eliminating low-gradient structures. Unlike other edge-preserving smoothing methods, it globally located important edges without local features. Therefore, a small number of important edges with small gradients are often removed. In 2017, Badshah and Shah [13] propose a novel model for segmentation of texture images based on gradient minimisation. It uses the image smoothing algorithm for the segmentation of texture images. In 2018, Ni and Wu [14] present an adaptive patched gradient minimisation model. However, the computational complexity of this method is very high. Blocking also inevitably leads to gaps between image patches. In 2012, Xu et al. [15] propose a image smoothing via relative TV (RTV). The new variation measures can effectively extract the main structural edges from textures, while RTV is not good enough for some small-scale details. In 2019, Zhao et al. [16] propose a local activity measurement of the clipped and normalised variance or standard deviation to better preserve structural. It adds its own work to RTV and uses it for image smoothing and denoising. Its smoothing effect is not good in complex texture regions. In 2013, Lopez-Molina et al. an anisotropic diffusion method using restricted dissimilarity functions. It replaces the absolute difference by a restricted dissimilarity function to measure the dissimilarity of the intensity two contiguous pixels, which determines the amount of diffusion that takes place at each position of the image. However, there are staircases in the smoothed image, and it needs to be iterated hundreds of times. In 2013, Li et al. [17] propose a method called edge-aware image manipulation. It processes a Gaussian pyramid from coarse to fine, and at each level, applies a non-linear filter bank to the neighbourhood of each pixel. However, it only supports two-scale manipulation. In 2014, Min et al. [18] propose an efficient technique based on [3] for performing a spatially inhomogeneous edge-preserving image smoothing, called fast global smoother (FGS). It combines two paradigms efficient edge-preserving filters and optimisation-based smoothing. However, it works badly in image with complex texture details. When the input data is very sparse, it takes a lot of time. In 2014, Zhang et al. [19] propose a method called rolling guided filter (RGF) to filter the images based on a joint bilateral filter. It removes the texture by Gaussian filtering and then recovers the edge of the input image by BLF. However, RGF may filter the non-texture component and cause ringing artefacts around edges. In 2015, Liu et al. [20] present an optimisation model making use of the redundancy of natural images, allowing details throughout the whole image to be removed automatically in a data-driven manner. It also introduces the norm and the norm as the regular term. It works well for image smoothing with regular textures, but it has a poor smoothing effect on images with irregular textures. In 2017, Kim et al. [21] propose a fast image smoothing algorithm. Combining existing work, the time complexity is reduced by optimising the algorithm. While speeding up the algorithm, there are residual texture details in complex texture regions. In 2018, Ma et al. [22] present a method for image smoothing combined with image decomposition and sparse high frequency gradient (DSG). It works well in removing complex textures. However, it removes some important structural edges while removing the texture details. In 2019, Zhu et al. [23] propose a benchmark for edge-preserving image smoothing. It constructs an image database and corresponding ground truth, but the main purpose is to combine deep learning with image smoothing. There are major conflicts between removing texture details and preserving structural edges. How to correctly distinguish textures and edges is the key to image smoothing. However, the image is often affected by shooting angle and light intensity during the acquisition process. So, the gradient at textures and edges can vary greatly, which making it more difficult to correctly distinguish texture details and structural edges. The structural edges can be well preserved via gradient minimisation. However, the texture details with larger gradients are also preserved. Image decomposition can reduce the gradient of the texture detail region, thus effectively smoothing the texture details with larger gradients. However, some important edge structures are also smoothed out due to the smaller gradient. Our method achieves the goal of increasing the gradient of structural edges while reducing the gradient of the texture details in order to correctly distinguish between texture details and structural edges. Based on this, we propose a combined image smoothing framework, which is a two-stage image smoothing method based on edge-patches equalisation and patch decomposition. 1. An image segmentation method based on image edge distribution is proposed for finding the region where the structural edges (or texture details) are concentrated. First, fast segmenting the input image. Then, extracting the edge map of input image. Finally, readjusting the boundary lines in a certain interval based on the edge map for removing seams. 2. Equalising and inversely equalising the edge-patch with histogram. In order to reasonably increase the gradient of structural edges, the edge-patch is equalised by the histogram. In order to ensure the continuity of boundary pixels, the smoothed edge-patch is inversely equalised. 3. Decomposing each patch. Reducing the gradient of texture details on the premise that the structural edges' gradient decrease less, so that the texture details and the structural edges can be more easily distinguished. 4. A two-stage image smoothing method via gradient minimisation is proposed. Each patch is smoothed via gradient minimisation separately for removing mail texture details. The whole image is fast smoothed via gradient minimisation for removing residual texture details and seams. The structure of this paper is organised as follows. The existing problems are described in Section 2. Section 3 illustrates the method we proposed. Section 4 is the comparison of the method in this paper and the exist advanced methods. Several applications and related examples of image smoothing are described in Section 5. Section 6 is the summary of work and the next work. 2 Problem Xu et al. [12] proposed an image smomethod via gradient minimisation. For simplicity of description, we will use ' smoothing' instead of it. smoothing can smooths the image while effectively retaining the important structural edges, but the texture details with large gradients are also preserved, as shown by the blue lines in Figs. 1b and 2b. In order to solve this problem, Liu et al. [20] propose an image smoothing algorithm based on image decomposition and sparse high frequency gradient. For simplicity of discription, we will use 'DSG' instead of it. DSG can effectively remove texture regions with a large gradient. Since the gradients of some important structural edges become smaller over image decomposition, they are removed as texture details, as shown by the red lines in Figs. 1c and 2c. Fig. 1Open in figure viewerPowerPoint Problems in the horizontal direction (a) Input image, (b) Smoothed image by smoothing, (c) Smoothed image by DSG, (d) Ordinate of the black curve represents r channel grey value in (a) marked with a red line. Ordinate of the blue curve represents r channel grey value in (b) marked with a red line. The ordinate of red curve represents r channel grey value in (c) marked with a red line, (e) Ordinate of the black curve represents r channel grey value in (a) marked with a blue line. Ordinate of the blue curve represents r channel grey value in (b) marked with a blue line. Ordinate of the red curve represents r channel grey value in (c) marked with a blue line Fig. 2Open in figure viewerPowerPoint Problems in the vertical direction (a) Input image, (b) Smoothed image by smoothing, (c) Smoothed image by DSG, (d) Ordinate of the black curve represents r channel grey value in (a) marked with a red line. Ordinate of the blue curve represents r channel grey value in (b) marked with a red line. Ordinate of red curve represents r channel grey value in (c) marked with a red line, (e) Ordinate of the black curve represents r channel grey value in (a) marked with a blue line. Ordinate of the blue curve represents r channel grey value in (b) marked with a blue line. Ordinate of the red curve represents r channel grey value in (c) marked with a blue line Figs. 1d, e, 2d and e are line charts for visually comparing the advantages and disadvantages of smoothing and DSG. Figs. 1d (2d) show r channel grey changes of the red row lines in Figs. 1a–c (column lines in Figs. 2a–c) containing important structural edges. By comparing Figs. 1d and 2d, it can be seen that smoothing can preserve the important edges of the image, but the DSG will remove some important edges. Figs. 1e (2e) shows the r channel grey changes of the blue row lines in Figs. 1a–c (column lines in Figs. 2a–c) containing complex texture details. By comparing Figs. 1e and 2e, it can be found that the DSG can smooth complex texture details, but the result of the smoothing has residual texture details. Based on the above problems, this paper proposes a two-stage image smoothing method based on edge-based image patch histogram equalisation and patch decomposition. 3 Description of proposed method Due to the problem that the gradient value of the same structural edges or texture detail varies greatly in different regions, a two-stage image smoothing algorithm based on edge-based image patch histogram equalisation and patch decomposition is proposed. The new method includes the following processes: (1) Segmenting the image and readjusting the boundary lines of the image patches are based on the distribution of the edges. (2) Equalising and inversely equalising the edge-patches by the histogram. (3) Decomposing each patch and extracting the smooth component. (4) Smoothing each patch an the whole image by smoothing. 3.1 Edge-based image segmentation The images are affected by the shooting distance and angle as well as the light intensity in the acquisition process. As a result, the gradient of the same edge (or texture) in different regions of the image is significantly different. Some important structural edges are considered texture details because of the small gradients, which are removed during image smoothing. In order to find the patches containing more structural edges, the images need to be segmented. Patches with more structural edges require histogram equalisation. Since the equalisation process can effectively increase the gradient of the structural edges whose gradient become smaller for various reasons, these important structural edges are preserved in the subsequent smoothing process. Image segmentation also produces seams between image patches (shown in Fig. 3a). The more patches are, the more seams there are, the more seams remaining in the smoothed image. At the same time, the histogram equalisation technique produces more seams. In order to preserve the important structural edges and avoid the seams, the number of image patches should not be excessive. Therefore, the size of the image patch is determined by the number of image patches. The side length of the image patch is defined as follows: (1)where W and H are the width and height of the input image, respectively. r is the side length of the image patch. represents the rounding operation. Fig. 3Open in figure viewerPowerPoint The histogram equalisation and inverse histogram equalisation (a) Smoothed image that edge-patches are equalised by the histogram and each patch is smoothed, (b) Edge-patches in (a) are inversely equalised, (c) Smooth image without histogram equalisation, (d) Smooth image after histogram equalisation After the image is segmented, the boundary line between image patches may pass through the region where structural edges are concentrated, so that many important edges are simultaneously divided into different image patches. Therefore, the boundary lines need be readjusted. Edge map is got by Canny edge detection [24] (edge(X, 'Canny', 0.55) in MATLAB). In order to get more edges, the sigma is set to 0.55. Then calculating the sum of edge pixels in each row (or column). Finally, finding the row (or column) containing the least edge pixels as the new boundary line in . The new horizontal boundary line is calculated as follows: (2)where represents the rounding operation, and E is the binary edge map. is the old horizontal boundary line. Then, the new vertical boundary line is (3)where is the old vertical boundary line. 3.2 Histogram equalisation and inverse histogram equalisation of edge-patch Image segmentation and histogram equalisation causes the pixels on patch boundary to be discontinuous, resulting in seams between patches. And the larger the size of patches is, the less the seams are. Therefore, the region of the edge-patch is enlarged when it is equalised by the histogram. The side length of edge-patch is added , so the size of new edge-patch is . It should be noted that only the middle region is reserved for ensuring each pixel has a unique value. 3.2.1 Histogram equalisation In Section 3.1, we get the edge map of the input image. So, we can calculate the edge pixel rate R of the input image. The formula is (4)where is the number of edge pixels of the input image. W and H are the width and height of the input image, respectively. Next, the edge pixel rate of each patch is calculated by the following equation: (5)where is the number of edge pixels and is the size of each patch. If is greater than R, the patch is an edge-patch; otherwise, it is the non-edge-patch. The edge-patch contains a wealth of important structural edges, but the gradient of partial important structural edges is too small for various reasons. So they are often smoothed out as texture details. Histogram equalisation can effectively increase the gradient of pixels in these important structural edges, while the gradient of the texture details increases less. The new value of pixel with a value of k is (6)where k () is the old value of pixel in the edge-patch, and is the number of pixels whose value is j. represents a rounding operation. 3.2.2 Inverse histogram equalisation After the histogram equalised edge-patches are smoothed, the important structural edges are well preserved while removing the texture details. However, there are significant differences between the equalised edge-patches and the unequalised non-edge-patches (as shown in Fig. 3a). In order to ensure the consistency of pixels in the whole image, it is necessary to inverse equalise the edge-patches. The new value of pixel with a value of k is calculated (7)where is the number of pixels whose value is k, and is the jth pixel whose value is k. is used to extract the coordinates of in the edge-patch and convert it to the coordinates in the input image. X is the input image and represents the value of pixel at position in X. Fig. 3 shows the histogram equalisation and inverse histogram equalisation in a more intuitive way. Fig. 3a is the image that edge-patches are equalised by the histogram and each patch is smoothed. The edge-patches in Fig. 3a are inversely equalised to get Fig. 3b. Fig. 3c is the smooth image without histogram equalisation. Fig. 3d is the smooth image after histogram equalisation. By comparing Figs. 3a and b, it can be seen that inverse equalisation can preserve the continuity of pixels at the boundary. By comparing Figs. 3c and d, it can be seen that equalisation can preserve some important structural edges. 3.3 Image patch decomposition The existing smoothing methods have a good effect on simple texture regions (the gradients are smaller), but there are texture residues in complex texture regions (the gradients are large). Therefore, reducing the overall gradients of complex texture regions is the key to improving the smoothing effect. In image super-resolution [25, 26] and completion [26, 27], an image can be decomposed into smooth components and non-smooth components. Therefore, we can get an image (smooth components) with small gradients by decomposing image patch x with the gradients as constraints. To extract the smooth components of x, the following optimisation problems need to be solved: (8)where x is the input patch. is a low-pass filter with all values set to 1. is the convolution operator. is controls the smoothness of . are the gradient operator along directions . This problem can be efficiently solved by FFT (fast Fourier transform) operator (9)where is the FFT operator and is the inverse FFT operator. is the complex conjugate operator, is the component-wise multiplication and is the component-wise division. So, is the smooth component of the input patch x. is the non-smooth component that contains the high frequency informations (mainly contains texture details in this paper) of x. Fig. 4 shows the gradient maps of with for different . Experiments show that the larger the is, the smaller the gradient of the texture region is. However, the gradient of the important structural edge has also become smaller and smaller. When , it can preserve the gradients of the structural edges while reducing the gradient of the texture details. Fig. 4Open in figure viewerPowerPoint When , the gradient maps of obtained by different , (a) No decomposition, (b) , (c) , (d) Figs. 5b–f show the non-smooth components with for different . It can be seen that the larger the value of is, the richer the high frequency information is. When is less than or equal to 10, the texture details are more than structural edges. However, edge information is increasing as becomes larger. In this paper, the value of is set to 40. Fig. 5Open in figure viewerPowerPoint The effect of (a) Undecomposed image. When , the non-smooth component for different , (b) , (c) , (d) , (e) , (f) 3.4 Two-stage smoothing We have increased the gradient of the structural edges as much as possible while reducing the gradient of the texture details. After the previous work, it is possible to distinguish the structural edges and texture details by the gradient. Therefore, we uses the norm to constrain the sparse high frequency gradient for smoothing the image patches and the whole image. (10)where is the smooth component in Section 3.3, and is the smoothing image. is the smoothing factor. The larger the value is, the fewer edges in are. Fig. 6 shows the smoothing image under different . The mesh texture remains in Figs. 6a–c, and the edges in Figs. 6d–f are more blurred. In order to control the trade-off between the textures and edges, the value of is set to 0.035. represents the number of pixels whose gradient is not equal to 0, which is defined as follows: (11)where p denotes the label of each pixel in , denotes the pixel labelled p in , and N denotes the total number of pixels in . is the gradient of along the horizontal direction (x), and is the gradient of along the vertical direction (y). denotes the number of pixels whose gradient is not equal to 0. Fig. 6Open in figure viewerPowerPoint When , the results for different (a) , (b) , (c) , (d) , (e) , (f) Equation (11) is a -norm regularisation optimisation problem. However, it is known that the -norm regularised optimisation problem is computationally intractable. A splitting method that iteratively optimises subproblems alternately [28] can be used as an effective technique, and each sub-problem can obtain a closed solution. Introducing the auxiliary variable , corresponds to and , respectively. So the function is redefined as (12)where is an automatic adapting parameter which is used to control the similarity between the variable (h, v) and its corresponding gradient. The larger the is, the more texture in is. Fig. 7 shows the smoothing image under different with . There is mesh texture in Figs. 7c–f. In order to control the trade-off between the textures and edges, the value of is set to 2. Fig. 7Open in figure viewerPowerPoint When , the results for different (a) , (b) , (c) , (d) , (e) , (f) Equation (12) can be divided into two sub-problems iteratively optimised. In each iteration, a set of variables is fi