Image denoising by low‐rank approximation with estimation of noise energy distribution in SVD domain

IET Image ProcessingVolume 13, Issue 4 p. 680-691 Research ArticleFree Access Image denoising by low-rank approximation with estimation of noise energy distribution in SVD domain Linwei Fan, Linwei Fan orcid.org/0000-0001-9986-2396 School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, BinHaiZhong Road No. 191, Yantai, People's Republic of ChinaSearch for more papers by this authorRan Meng, Ran Meng School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of ChinaSearch for more papers by this authorQiang Guo, Qiang Guo School of Computer Science and Technology, Shandong University of Finance and Economics, Bicyclic East Road No. 7366, Jinan, People's Republic of ChinaSearch for more papers by this authorMiaowen Shi, Miaowen Shi School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of ChinaSearch for more papers by this authorCaiming Zhang, Corresponding Author Caiming Zhang czhang@sdu.edu.cn School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, BinHaiZhong Road No. 191, Yantai, People's Republic of China Shandong Province Key Laboratory of Digital Media Technology, Bicyclic East Road No. 7366, Jinan, People's Republic of ChinaSearch for more papers by this author Linwei Fan, Linwei Fan orcid.org/0000-0001-9986-2396 School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, BinHaiZhong Road No. 191, Yantai, People's Republic of ChinaSearch for more papers by this authorRan Meng, Ran Meng School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of ChinaSearch for more papers by this authorQiang Guo, Qiang Guo School of Computer Science and Technology, Shandong University of Finance and Economics, Bicyclic East Road No. 7366, Jinan, People's Republic of ChinaSearch for more papers by this authorMiaowen Shi, Miaowen Shi School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of ChinaSearch for more papers by this authorCaiming Zhang, Corresponding Author Caiming Zhang czhang@sdu.edu.cn School of Computer Science and Technology, Shandong University, ShunHua Road No. 1500, Jinan, People's Republic of China Shandong Co-Innovation Center of Future Intelligent Computing, BinHaiZhong Road No. 191, Yantai, People's Republic of China Shandong Province Key Laboratory of Digital Media Technology, Bicyclic East Road No. 7366, Jinan, People's Republic of ChinaSearch for more papers by this author First published: 07 March 2019 https://doi.org/10.1049/iet-ipr.2018.6357Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Low-rank approximation has shown great potential in various image tasks. It is found that there is a specific functional relationship about singular values between the original image and a series of noisy images, which can be used to construct the singular values of a noise-free image. In this study, the authors propose a novel denoising method based on the above facts and low-rank approximation theory. Firstly, they estimate the noise energy distribution of the group matrix in the singular value decomposition (SVD) domain using the energy characteristics of the image with different noise levels. The energy distribution of the noise is shrunk to obtain the energy distribution of the true signal. Then, based on the optimal energy compaction property of SVD, the low-rank property of matrix is constrained in the SVD domain to obtain the low-rank approximation of the matrix. Moreover, an iterative back projection method is adopted in this study to suppress residual noise. A new noise standard deviation estimation approach, targeted at the back projection process, is proposed to effectively optimise the denoising results during the iteration. Experimental results show that the authors' method efficiently decreases the noise and achieves comparable denoising performance to the state-of-the-art methods regarding both quantitative measurement and visual effect. 1 Introduction During acquisition and transmission, images are inevitably contaminated by noise, which degrades image quality and seriously affects the subsequent processing, such as image segmentation, feature extraction and pattern recognition. Therefore, image denoising has always been one of the hotspots in image processing. In this paper, we focus on additive white Gaussian noise (AWGN) [1]. The denoising problem can be represented as follows: (1)where y is the observed noisy image, x is the clean image, and n represents the AWGN with standard deviation which can be estimated by various methods such as median absolute deviation [2], block-based estimation method [3] and PCA-based ones [4]. The aim of denoising algorithm is to remove the noise accurately while preserving the detail of texture and edge of image. Denoising methods consist of two categories in terms of the different disposal domains – spatial domain based methods and transform domain based methods. Spatial domain based methods exploit the relationship among pixels. Transform domain based methods process the transform coefficients in transform domain and then the solution is evaluated by the inverse transform of new coefficients. Among the traditional spatial domain methods, the most typical one is spatial filtering algorithm [5], which uses the continuity and similarity of pixels in the local area. However, such methods are less effective when the noise is stronger since noise destroys the correlation between pixels in the local area. Therefore, it's necessary to introduce the non-local self-similarity (NSS) of the image to consider the information characteristics of the whole image. Natural image contains many similar image patches whose positions are not limited to local regions and provide useful redundant information for image reconstruction. Buades et al. [1] proposed the non-local means (NLM) filtering, which estimates each pixel by the non-local average of all the pixels in the image. This influential work takes full advantage of the redundant information and stimulates extensive research on image restoration using NSS. Many variants of NLM have been developed [6-11] and methods exploiting the NSS have been proposed from the spatial domain to the transform domain. For example, based on the NSS property of an image, Fan et al. [11] proposed an adaptive texture-preserving denoising method via combining the gradient histogram matching prior and the NSS prior. Transform domain methods assume that the image can be sparsely represented in the transform domain [12], such as wavelet transform domain and discrete cosine transform domain [13-16]. Due to the sparsity of representation coefficient, the main information of the image is concentrated on the few larger coefficients. Therefore, noise can be distinguished by different coefficient shrinkage methods. However, the fixed transform fails to provide an adaptive sparse representation for the complex structures. In order to overcome the problem, Aharon et al. [17] proposed a creative method called K-SVD (singular value decomposition) which learns an over complete dictionary to provide an adaptive sparse representation for image. Elad and Aharon [18] used the dictionary learnt in [17] to represent the image and remove noise by constraining the sparsity of coefficient. This method is a local method for calculating sparse coding of image patches independently. Zhang and Desrosiers [19] proposed a robust algorithm combining gradient histogram with a sparse representation to obtain good estimates of the sparse coding coefficients of the latent image and realising image denoising while preserving the texture. Dabov et al. [20] combined the NSS with transform domain method and proposed the block-matching and sparse 3D collaborative filtering (BM3D), which groups non-local similar 2D image blocks into 3D data arrays and denoises utilising the sparsity of coefficient after wavelet transform. Despite that extensions of BM3D were proposed in the aspect of colour image denoising and video processing [21-23], wavelet transform cannot provide an adaptive representation. Sparse coding denoising method combining NSS and adaptive representation also shows a certain advantage, for example learned simultaneous sparse coding [24] and non-locally centralised sparse representation [25]. Liu [26] proposed an adaptive sparse coding on PCA dictionary, which learns a dictionary using PCA basis [27] and denoises by adaptive sparse coding. Inspired by the success of the sparse representation, group-based sparse coding has attracted considerable interests in image denoising [28-30]. The key to constructing the denoising method is how to make full use of the prior information in image. More recently, low-rank approximation [31, 32] for extracting low-dimensional structures in data has attracted attention in image denoising resulting in an explosion of research by the approach. The basic idea of low-rank matrix approximation (LRMA) is to approximate noise-free image patches by low-rank modelling of image non-local similarities. In other words, the image patches are grouped by block matching, such that the patches in each group share similar underlying image structures. Thus, stack of these similar patches into a matrix, with vectorised image patches as its columns, forms a noisy version of an approximately low-rank matrix. Therefore, the task of denoising is recovering the underlying low-rank matrix from its noisy observation. Cai et al. [33] proposed that LRMA can be achieved by nuclear norm minimisation (NNM) and suggested to solve NNM by soft-thresholding of singular values. Gu et al. [34, 35] proposed a weighted NNM (WNNM) model to adaptively assign different weights to singular values since the largest singular values should be shrunk less. However, the denoising algorithm requires multiple iterations to achieve a considerable result. Dong et al. [36] applied the energy compaction property of SVD to sparse representation, which gives singular values a physically plausible interpretation from the bilateral variance estimation point of view. Guo et al. [37] proposed a low-rank approximation SVD (LRA-SVD) method which takes advantage of the optimal energy compaction property of SVD to estimate the matrix by only retaining a certain number of the largest singular values, thus giving the low-rank approximation of noisy matrix. With the appropriate estimation of rank, LRA-SVD reaches a competitive result with state-of-the-art algorithms in the two-stage approach. However, the method of truncating singular values assumes that the energy of real information concentrates on the first few large singular values completely while the noise concentrates on the rest. This assumption ignores that the energy compaction of SVD is not strict and the energy of the noise also spreads among the largest singular values. What's more, the performance of two-stage approach is poor when the noise level is high. In view of the above problems, in this paper, we propose a denoising method to estimate the matrix of similar patches by low-rank approximation in SVD domain and combined with shrinkage and truncation of singular values. The innovations of this paper are summarised as follows: (i) Firstly, by using the energy characteristic of images with different noise levels, the energy distribution of real signal in the SVD domain is estimated as a benchmark of singular value shrinkage. Then, the largest singular values are preserved to approximate the matrix with low rank according to the energy compaction property of SVD. (ii) We consider the singular structures such as edges while researching similar patches, and employ the edge characteristics to control the computation. (iii) Ameliorating the estimation of the noise coefficient in the iterative back-projection process to make the result of next iteration more accurate. Experimental results show that the proposed method outperforms state-of-the-art denoising methods not only in quantisation index, but also in local structure preservation, leading to more pleasant visual effect. The remaining of this paper is organised as follows. Section 2 reviews the SVD. Section 3 presents the details of the proposed method. The experimental results are reported in Section 4. Finally, we conclude the paper in Section 5. 2 Review of SVD and related denoising methods 2.1 Denoising by low-rank approximation in SVD domain According to [33, 38, 39], the SVD of the matrix is one of the most valuable methods in solving the low-rank approximation problems. For a noisy image, each group of similar image patches is vectorised into column vectors to form a composite matrix (). is the clear version of , which is made up of the corresponding patches of from clear image x. There are underlying similar image structures in so the columns of have high correlation and X has the low-rank property. This low-rank characteristic of is demonstrated in Fig. 1 by displaying the singular values of group matrices in noise-free Lena image. The singular values of low-rank matrices drop sharply, and the main information of matrices can be calculated by the first few larger singular values and corresponding singular vectors. Fig. 1Open in figure viewerPowerPoint Singular values of group matrices from image Lena In the theory of SVD, the real-valued low-rank matrix can be decomposed as (2)where num is the number of singular values and the is the singular values of whose diagonal entries are non-negative and in non-ascending order. and are unitary matrices with orthonormal column vectors. The vectors and are, respectively, the left and right singular vectors of corresponding to the singular value . has the same rank with , thus if the rank of is k, the diagonal entries satisfy that . can be considered as a result of the superposition of k subgraphs of rank 1, while singular value is the weight coefficient for the subgraph . In practice, under the AWGN model, the noisy observation can be obtained as (3)where the noisy matrix and the noise matrix are of full rank. With the addition of noisy matrix, the energy of noise will span the whole space in the SVD domain and the last singular values of will not be zero. The estimation of denoised matrix needs to solve the problem of approximating the noisy matrix with another matrix which has a specific low-rank k. The approximation is based on minimising the Frobenius norm of the difference between and under the constraint that . (4)It was proved in [39] that the above problem can be solved by the SVD of , namely (5)where trun is an operator which retains only the k largest singular values of and sets the rest to be zeros, i.e. (6)In general, according to the optimal energy compaction property of SVD, the energy of real signal mainly concentrates on the small amount of large singular values which correspond to the signal subspace. The small singular values, which usually correspond to the noise subspace, are truncated to zero to get the denoised low-rank matrix. The key issue is to determine the k to separate energy of noise from signal [40]. Guo et al. [37] employed a more efficient method to determine the rank k which is determined by the following method: (7)The truncation denoising method defined by (5)–(7) preserves the first k singular values and provides a hard-threshold solution, which ignores the energy of noise spreading in the first k values in SVD domain. As alternatives, soft-thresholding methods are suggested to determine the shrinkage of singular values [33, 34, 36], which consider the energy of noise spreading in the largest singular values. 2.2 Oracle denoising Rajwade et al. [41] have experimented and proposed that utilising the information of the noise-free patches can produce an ideal denoiser which far exceeds the state-of-the-art methods. Let be the noise-free version of , which is made up of the corresponding patches of from an image x. Given the SVD of : , the best singular values to estimate can be recovered by the projection of onto the base which is given as (8)However, the diagonal entries of can be negative or in non-decreasing order, which seriously violates the property of singular values. We take advantage of the singular values of (), the denoised version of can be estimated by (9)The singular value sequence reflects the energy distribution of the real signal in the SVD domain. Using this energy distribution yields a much better denoising result than using in (5). This experiment is impossible in practice since the noisy-free image is unavailable, however, it can serve as a benchmark and inspire us to manipulating the singular values of noisy matrix to approximate for better denoising result. 3 Proposed method The analysis of SVD implies that the noise can be better separated from the real signal in the SVD domain. Inspired by the previous works, we propose a new image denoising method. Our method exploits the energy distribution of noise in the SVD domain and the low-rank property of similar patch matrix to remove noise by considering both shrinkage and truncation of singular values. The proposed method begins with block matching and noisy image patches are grouped by thesimilarity measure based on the Euclidean distance. For singular structures, themethod extends the search region and uses an edge-assistive method to constrain thecalculation. The stack of matched patches are vectorised and re-arranged to form matrices.Then, each group of similar patches is regarded as a noisy low-rank matrix which canbe estimated in SVD domain. Finally, we obtain the denoised image by weightedaveraging all the processed patches. Fig. 2illustrates the procedure of the proposed method and the detail will be described inthe rest of this section. Fig. 2Open in figure viewerPowerPoint Flowchart of the proposed method 3.1 Edge information assistive block matching and grouping Owing to the natural redundancy of the image structures, similar patches grouping is commonly employed in NSS based method, and it is also the basic step in the proposed method. For simplicity, this paper uses block matching for grouping by the Euclidean distance. The noisy image y can be divided into M overlapping patches with size denoted by , where is the vectorised version of the patch at position i. We select patches in an image y with a step length smaller than s reference patches and the non-local similar patches of reference patch are searched in the search window. The Euclidean distance between two patches and is defined as (10)where is the candidate patch for in the search window. The smaller the is, the more similar and are, so we select num-1 vectorised patches with minimal distance from to construct a group matrix for the reference patch which is defined as (11)Since the similarity of the patches only considers the grey-level distance, it could be difficult to find enough similar patches in a limited search window around the edge regions which have less redundant information. In order to improve the search accuracy of similar patches, a size-adaptive search window is proposed in this paper. First, on the basis of the image gradient, all the pixels in the image are divided into edge and smooth regions by using Canny operator, and since the search for similar patches is to preserve the image features which are mainly located in edge regions, the size of the search window for the edge region can be extended. In the extended windows, the proposed method first searches in the edge image and only the patches having similar edge structures with the reference edge patch are selected to be candidate patches for searching in the noisy image. Fig. 3 illustrates the block matching in the extended search window, the black and white borders represent original search windows in edge image and noisy image, respectively, the blue borders represent the extended search window, the red blocks represent the reference patches and the green ones are edge matched patches in the extended search window. As shown in Fig. 3, there are edges in the reference patches, so the search windows are extended to the blue borders. Search the extended area in the edge image for patches having similar edge structures with the reference patch using filtering operation, which is time saving. The correlated patches of edge patches in noisy image are selected as candidate patches. Then the Euclidean distance between them and the reference patch is further calculated to improve the matching accuracy. Fig. 3Open in figure viewerPowerPoint Illustration of block matching in extended search windows 3.2 SVD-based denoising for patch matrix For each reference patch , a group matrix is formed for subsequent denoising. is made up of noisy patches and can be expressed as (12)where and are the corresponding noise-free matrix and noise matrix of , respectively. 3.2.1 Estimation of energy distribution of real signal The column vectors of are vectorised similar noise-free patches with high correlation, which leads to be a low-rank matrix. Fig. 4 illustrates the singular values of group matrices in image Lena with different standard deviations and demonstrates the energy distribution of signal. The noise-free matrix as a low-rank matrix has few large singular values, which means the information of noise-free signal is mainly concentrated on a small amount of large coefficients. In addition, the singular values change due to the addition of noise and the variations imply the energy distribution of additive noise in SVD domain. The singular values of different noise levels represent a specific functional relationship. Higher noise level leads to more increase of singular values. It has been shown that employing the singular values of noise-free matrix which better reflect the energy distribution of real signal will lead to an improved result [41], however, it is infeasible in practice due to the absence of noise-free matrix. Therefore, we proposed to estimate the singular values without noise based on the singular values of matrices with different noise levels. The functional relationship among singular values of matrices will be described in detail below. Fig. 4Open in figure viewerPowerPoint Singular values of grouped matrices in image Lena with the different noise levels For the noisy image y with AWGN of standard deviation , we introduce an interim image z which is formed by adding noise of standard deviation into y as follows: (13)z has a higher level of noise than y. Thus, is the matrix by adding noise into . The SVD of and are employed to get the singular values as follows: (14)Difference between and denotes the energy distribution of increased noise in than , which can be used to estimate the real energy distribution, that is the singular values of . The singular values of group matrices are displayed in Fig. 5 where each point is the ith singular values of the matrices from the noise-free image x (red), noisy image y (blue) and the interim image z (black), respectively. Singular value sequence of the noise-free matrix (red curve) represents the energy distribution of the real signal in SVD domain, meanwhile, the difference between it and (blue curve) represents the energy distribution of noise in . Fig. 5Open in figure viewerPowerPoint Singular values of group matrices from Lena In SVD domain, we assume that the energy distribution of noise in , marked by D, is positively correlated with the difference between and , i.e. . Therefore, we can obtain which is the approximation of real energy distribution in the noise-free matrix by shrinkage of as (15)Also, to maintain the non-ascending order of the diagonal entries in , D should be constrained by the difference of the diagonal entries in . Let i denotes the ith diagonal entry of the singular value matrix, D should satisfy the following objective function: (16)where is the proportional factor. The former of the formula leads the diagonal entries of D to be equal to times of difference between and . The latter constrains the difference of adjacent diagonal entries of D to keep non-ascending property of . It is necessary to adaptively change the shrinkage ratio according to the information contained in the image patch. High-frequency information (such as edges and textures) is more likely to be confused with noise, which is more seriously affected by noise when searching for similar patches. Thus, the energy distribution of the real signal is more dispersed in the SVD domain and a smaller is set for such matrices to avoid the loss of real information. On the contrary, the low-rank nature of the matrices in the flat area of the image is better. Also, the real information at low frequencies is not easily confused with high-frequency noise, and the disturbance of noise to singular values is more regular, so a larger can be set for such matrices. 3.2.2 Low-rank approximation with noise variance constraint The above processing only shrinks each singular value to make it more consistent with the distribution of the real signal, nevertheless, it does not perform low-rank processing. Hence, inspired by the previous method in Section 2.1, we estimate the denoised matrix of by (17)The parameter k, which denotes the rank of , guides the number of singular values will be retained. Let , then (18) is an approximation of with a bias (19)The error between and is calculated by the square summation over entries of , which is shown as (20)Since is the estimation of noise and the noise standard deviation is known to be . If the noise is estimated accurately, it should be satisfied that (21)By comparing (20) with (21), k can be determined as follows: (22)The denoising scheme defined in this section has the advantage over the truncated method in [37]. The shrinkage of singular values in (15) and (16) takes noise energy spreading in the largest singular values into account. In addition, the value of k defined by (22) makes the information removed from more aligned with the prior knowledge of noise. 3.3 Weighted combination Section 3.2 has denoised each group matrix formed in Section 3.1. In this section, we rearrange the columns of the matrix to patches and reconstruct the denoised image by weighted combining. Firstly, a patch may be allocated to several matrices in the grouping step, so we get the estimations of each patch by average. Moreover, the image patches are sampled with overlapping regions, which result in multiple estimations for each pixel. The value of each pixel in the image is determined by a weighted average of the estimated values from different patches, which will further suppress the noise. The definition rules of weight for each pixel are as follows. The similarity of grouped patches reflects the degree of correlation of columns in the group matrix. For a well-correlated matrix, the noise-free matrix will better concentrate on the low-dimensional subspace. Therefore, the higher the similarity between matrix columns, the more accurate the denoising result of the matrix. For a group matrix , the weight of can be denoted by (23)where Num is the number of pixels in . The ith pixel can be aggregated by (24)where M is the normalising constant, is the set of the index of all group matrices containing pixel and is the denoised estimation of the ith pixel in the jth matrix . 3.4 Iterative back projection Following the above steps, most noises can be removed. In this section, we use the iterative back projection process to reduce the residual noise and further improve the performance. Performance improvement of iterative back projection method has been expressed by wavelet-based model, low-rank based model and bilateral filtering model. The method has been widely used as a post-processing step. On the one hand, the removed noise may contain some real information so the re-injection of noise can reduce the loss of detail. The new noisy image generated by back projection, which contains more edges or textures than the denoised image. On the other hand, the estimated image with the noise feedback has a lower noise level than the original noisy image, which improves the accuracy of grouping and gradually optimises the denoising result. In a finite number of iterations, we can observe that the residual noise monotonically decreases and the structures of the image are improved. The basic idea of iterative back projection is to add estimate