# Block-Extraction and Haar Transform Based Linear Singularity Representation for Image Enhancement

Authors: Yingkun Hou; Xiaobo Qu; Guanghai Liu; Seong-Whan Lee; Dinggang Shen
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

In this paper, we develop a novel linear singularity representation method using spatial K-neighbor block-extraction and Haar transform (BEH). Block-extraction provides a group of image blocks with similar (generally smooth) backgrounds but different image edge locations. An interblock Haar transform is then used to represent these differences, thus achieving a linear singularity representation. Next, we magnify the weak detailed coefficients of BEH to allow for image enhancement. Experimental results show that the proposed method achieves better image enhancement, compared toblock-matching and 3D filtering(BM3D),nonsubsampled contourlet transform (NSCT), andguided image filtering.


---

## Body

## 1. Introduction

Image enhancement plays an important role in image processing and pattern recognition. Image enhancement techniques can be generally divided into two categories:(1) spatial methods and (2) frequency domain methods. Because frequency domain-based methods can represent image details in high-frequency subbands by certain transformations, the image enhancement can be achieved by magnifying the weak detailed transform coefficients.The contours and textures provide the most important information in most natural images and usually present linear singularities. Because a classical orthogonal wavelet transform can only effectively represent point singularities [1, 2], to better represent linear singularities, a series of beyond wavelets (i.e., Ridgelet [3, 4], Curvelet [5], Contourlet [6], and NSCT [7]) have been developed. However, either orthogonal wavelets or beyond wavelets will always directly convolute an image by a certain convolution kernel. Due to the convolution operation in filtering, artifacts are inevitably introduced after the inverse transform. In addition, to better represent linear singularities, more and larger support directional filtering banks have to be used, at the cost of increasing computational complexity.The convolution operation in traditional transforms is often performed between a kernel and a local neighborhood of the image. Therefore, these methods are called local methods. In recent years, some nonlocal image processing methods have been developed for image denoising, i.e., nonlocal-means (NL_means) [8] and block-matching and 3D filtering (BM3D) [9–15]. These nonlocal methods attempt to find some of the most similar image blocks, i.e., by implementing block-matching and the weighted means on the blocks [8], or by implementing a 3D transform on a 3D array stacked by similar blocks for enhanced sparsity representation and then for hard-threshold shrinkage on the transformed coefficients to achieve image denoising [9, 10, 16–19]. Because NL-means and BM3D were both initially developed to achieve image denoising, the sufficient similar image blocks are crucial for denoising. However, overly strong similarities will weaken the representation of differences among blocks in the 3D transformation, or, equivalently, the linear singularity in each block. This is problematic for applications such as image enhancement.Some new methods of image enhancement have been proposed recently. For example, Heet al.[20, 21] proposed a novel guided image filtering method for image enhancement. Wanget al.[22] proposed a color face image enhancement method using adaptive singular value decomposition in Fourier domain for face recognition. Gaoet al.[23] proposed an image enhancement method specifically for visual impairments. Liet al. [24] proposed an adaptive fractional calculus of small probability strategy-based method for achieving image denoising and enhancement. These methods usually enhance the background or have a halo phenomenon as well when used to enhance image details.In this paper, we propose a novel linear singularity representation method that avoids finding sufficient similar image blocks. The proposed method is based on an observation of natural images that depict many image blocks (or patches) with similar smooth backgrounds in a small neighborhood. Suppose that the detail (i.e., textures or contours) locations scarcely differ between two isometric image blocks that are both extracted from the same small local neighborhood, simply by subtracting one block by one block. The image details can then be effectively represented; for example, the details can be preserved only on relatively greater magnitudes, by setting all other magnitudes approximately equal to zero. Figure1 illustrates this situation. Inspired by this fact, we propose a novel linear singularity representation method, calledK-neighbor block-extraction and Haar transform(BEH). The Haar transform is chosen because of its low computational complexity but also because of its ability to represent sudden transitional signals. Although it lacks continuity and differentiability, this property is actually advantageous for analyzing signals with sudden transitions [25].Figure 1
An illustration using the proposed linear singularity representation method.In the proposed method, we first select an image block using a sliding window and then extract its spatial K-neighbor image blocks. All of the blocks extracted using this approach will have similar smooth backgrounds but with subtle differences in their detail locations. Next, we implement a fast Haar transform, by calculating either the weighted summation or subtraction among these blocks. Thus, the linear singularity can be effectively represented. To verify the effectiveness of our proposed method, we apply it to image enhancement in order to improve the visibility of images, which is crucial for image processing and computer vision [26–31]. Experimental results demonstrate that the proposed method achieves performance in image enhancement that is superior to that of the existing state-of-the-art linear singularity representation methods, including nonsubsample contourlet transform (NSCT).


## 2. NSCT and BM3D for Linear Singularity Representation in Image Enhancement

### 2.1. Linear Singularity Representation

Because orthogonal wavelets can only effectively represent point singularities, a series of beyond wavelets has been developed for linear singularity representation. Among them, the NSCT has some favorable properties, such as translation invariability and multidirectional filtering, which makes it one of the state-of-the-art linear singularity representation methods. Although NSCT achieves the best linear singularity representation performance, it has drawbacks similar to those of other local transform methods, i.e., it always introduces some artifacts after a certain type of operation on the transformed coefficients and inverse transform. The introduction of artifacts comes from the convolution operation of the inverse transform,(1)fx,y=∬R2Ca,bΨ-1a,bdadbwhere C is the transformed coefficients, Ψ-1 is the inverse transform filtering bank, and f(x,y) is the reconstructed image. For image denoising, if one implements a hard thresholding operation, some of the isolated noisy points may remain after the inverse transform. As these noisy points have influences on their surrounding pixels, the ringing artifacts for orthogonal wavelets and the strip artifacts for beyond wavelets will be introduced. An illustrated example of this situation is shown in Figure 2. From this figure, we can see that some strip artifacts are introduced when enhancing some coefficients of a certain directional subband of NSCT.Figure 2
An illustration of strip artifacts by NSCT. (a) A directional subband of NSCT and some enhanced coefficients. (b) The reconstructed image by the inverse NSCT. (c) A fragment of the original image. (d) A reconstructed fragment of (c) by the inverse NSCT.
(a) (b) (c) (d)Among recently developed nonlocal methods, BM3D is one of the state-of-the-art image denoising methods [32]. Specifically, exploring other nonlocal approaches to achieve ideal image denoising performance resulted in the creation of BM3D. In particular, BM3D does not consider linear singularity representation whatsoever, which may cause problems to surface with the use of the block-matching technique. In the 3D transform, only the 2D transform on each image block can be seen as a singularity representation. However, this singularity representation degenerates the general 2D wavelet transform method because the third-dimensional transform is on considerably similar image blocks, which implies that singularity cannot be represented by the Haar transform. In the extreme case, if two image blocks are completely alike, the subtraction between the two blocks will equal zero, and there will be no information in the high-frequency subbands of the Haar transform. There are always some differences among image blocks in a group formed by block-matching, even though the ideal linear singularity representation cannot be achieved. So, to achieve better linear singularity representation performance, some local differences must exist among a group of blocks. Considering this, we propose to modify the block-matching in the BM3D method to be block-extraction.In the BM3D method, the block-matching is implemented by computing the Euclidean distances between a given reference block and each block in the neighborhood of the reference block,(2)D=T2DZR-T2DZ22N12where ZR is the reference block, Z is the block to be matched, and T2D is a 2D transform, i.e.,eitherDCToran orthogonal wavelet transform. However, this 2D transform does change the block-matching results corresponding to the case of using the original image blocks to calculate distance in block-matching. N1 is the size of each square image block. Because T2D does not affect block-matching, we simplify (2) to be the following one:(3)D=ZR-Z22N12Here, we can show a block-matching example by (3) and give the 3D transform results of the grouped image blocks in Figure 3. From the transformed results in Figure 3, we can see that there is scarce information in the high-frequency subbands. The main reason is that all of the blocks are too similar in a group; that is, there is not enough singularity among these blocks.Figure 3
Demonstration of block-matching and 3D filtering. (a)-(b) The result of block-matching in a neighborhood of House image. (c) The result of 3D transform on the 3D array formed by all of the blocks in (b).


### 2.2. Application to Image Enhancement

#### 2.2.1. Image Enhancement by NSCT

NSCT has excellent linear singularity representation performance, which allows some relatively weak edges in images to be better represented by NSCT than by BM3D. Therefore, when using NSCT to achieve better image enhancement, strong edges and weak edges should be processed differently. In [7], to achieve better image enhancement results, the definition below is used to differentiate among strong edge, weak edge, and noise: (4)strongedge,ifmean≥cσweakedge,ifmean<cσ,max≥cσnoise,ifmean<cσ,max<cσwhere c is a parameter ranging from 1 to 5 and σ is the standard deviation of noise in the subbands at a specific pyramidal level. In this case, a NSCT-based image enhancement algorithm can be given as follows:(5)yx=x,strongedgepixelsmax⁡cσxp,1x,weakedgepixels0,noisepixelswhere the input x is the original transform coefficient and 0<p<1 is the amplifying gain. This function keeps the coefficients of strong edges, amplifies the coefficients of weak edges, and zeros the noise coefficients.


#### 2.2.2. Image Enhancement by BM3D

According to the analysis in Section2.1, BM3D cannot obtain the linear singularity representation like NSCT. Therefore, BM3D cannot use the same algorithm as NSCT to achieve image enhancement. In [33], a joint image enhancement and denoising algorithm by 3D transform-domain collaborative filtering was proposed. In this algorithm, block-matching and 3D transform were implemented. Additionally, the hard thresholding operation on the 3D transformed coefficients was used to remove noise, and the alpha-rooting method was used to amplify coefficients and achieve image enhancement.Given a transform spectrumt of a signal, which contains a DC coefficient termed t(0), the alpha-rooting is performed as (6)tshi=signti∙t0∙tit01/α,ift0≠0ti,otherwisewhere tsh is the spectrum of the transformed signal and an exponent α>1 leads to the enhancement of image details.In [33], two different approaches are given to perform image enhancement: BM3D-SH3D and BM3D-SH2D. The first one implements the alpha-rooting algorithm on the 3D transformed spectrum, while the second implements the alpha-rooting algorithm only on the 2D transformed spectrum after the 1D inverse transform, i.e., on each block-transformed spectrum. However, actually, both methods achieve image enhancement only via one block in a certain sense; because of extreme similarity among the blocks in a group, the magnitudes of the coefficients in the high-frequency subbands are all nearly zero.Accordingly, to achieve better enhancement on image details, we propose using a K-neighbor block-extraction method to replace the block-matching procedure in BM3D and further discard the 2D transform on each image block by only implementing an interblock 1D Haar transform.


## 2.1. Linear Singularity Representation

Because orthogonal wavelets can only effectively represent point singularities, a series of beyond wavelets has been developed for linear singularity representation. Among them, the NSCT has some favorable properties, such as translation invariability and multidirectional filtering, which makes it one of the state-of-the-art linear singularity representation methods. Although NSCT achieves the best linear singularity representation performance, it has drawbacks similar to those of other local transform methods, i.e., it always introduces some artifacts after a certain type of operation on the transformed coefficients and inverse transform. The introduction of artifacts comes from the convolution operation of the inverse transform,(1)fx,y=∬R2Ca,bΨ-1a,bdadbwhere C is the transformed coefficients, Ψ-1 is the inverse transform filtering bank, and f(x,y) is the reconstructed image. For image denoising, if one implements a hard thresholding operation, some of the isolated noisy points may remain after the inverse transform. As these noisy points have influences on their surrounding pixels, the ringing artifacts for orthogonal wavelets and the strip artifacts for beyond wavelets will be introduced. An illustrated example of this situation is shown in Figure 2. From this figure, we can see that some strip artifacts are introduced when enhancing some coefficients of a certain directional subband of NSCT.Figure 2
An illustration of strip artifacts by NSCT. (a) A directional subband of NSCT and some enhanced coefficients. (b) The reconstructed image by the inverse NSCT. (c) A fragment of the original image. (d) A reconstructed fragment of (c) by the inverse NSCT.
(a) (b) (c) (d)Among recently developed nonlocal methods, BM3D is one of the state-of-the-art image denoising methods [32]. Specifically, exploring other nonlocal approaches to achieve ideal image denoising performance resulted in the creation of BM3D. In particular, BM3D does not consider linear singularity representation whatsoever, which may cause problems to surface with the use of the block-matching technique. In the 3D transform, only the 2D transform on each image block can be seen as a singularity representation. However, this singularity representation degenerates the general 2D wavelet transform method because the third-dimensional transform is on considerably similar image blocks, which implies that singularity cannot be represented by the Haar transform. In the extreme case, if two image blocks are completely alike, the subtraction between the two blocks will equal zero, and there will be no information in the high-frequency subbands of the Haar transform. There are always some differences among image blocks in a group formed by block-matching, even though the ideal linear singularity representation cannot be achieved. So, to achieve better linear singularity representation performance, some local differences must exist among a group of blocks. Considering this, we propose to modify the block-matching in the BM3D method to be block-extraction.In the BM3D method, the block-matching is implemented by computing the Euclidean distances between a given reference block and each block in the neighborhood of the reference block,(2)D=T2DZR-T2DZ22N12where ZR is the reference block, Z is the block to be matched, and T2D is a 2D transform, i.e.,eitherDCToran orthogonal wavelet transform. However, this 2D transform does change the block-matching results corresponding to the case of using the original image blocks to calculate distance in block-matching. N1 is the size of each square image block. Because T2D does not affect block-matching, we simplify (2) to be the following one:(3)D=ZR-Z22N12Here, we can show a block-matching example by (3) and give the 3D transform results of the grouped image blocks in Figure 3. From the transformed results in Figure 3, we can see that there is scarce information in the high-frequency subbands. The main reason is that all of the blocks are too similar in a group; that is, there is not enough singularity among these blocks.Figure 3
Demonstration of block-matching and 3D filtering. (a)-(b) The result of block-matching in a neighborhood of House image. (c) The result of 3D transform on the 3D array formed by all of the blocks in (b).


## 2.2. Application to Image Enhancement

### 2.2.1. Image Enhancement by NSCT

NSCT has excellent linear singularity representation performance, which allows some relatively weak edges in images to be better represented by NSCT than by BM3D. Therefore, when using NSCT to achieve better image enhancement, strong edges and weak edges should be processed differently. In [7], to achieve better image enhancement results, the definition below is used to differentiate among strong edge, weak edge, and noise: (4)strongedge,ifmean≥cσweakedge,ifmean<cσ,max≥cσnoise,ifmean<cσ,max<cσwhere c is a parameter ranging from 1 to 5 and σ is the standard deviation of noise in the subbands at a specific pyramidal level. In this case, a NSCT-based image enhancement algorithm can be given as follows:(5)yx=x,strongedgepixelsmax⁡cσxp,1x,weakedgepixels0,noisepixelswhere the input x is the original transform coefficient and 0<p<1 is the amplifying gain. This function keeps the coefficients of strong edges, amplifies the coefficients of weak edges, and zeros the noise coefficients.


### 2.2.2. Image Enhancement by BM3D

According to the analysis in Section2.1, BM3D cannot obtain the linear singularity representation like NSCT. Therefore, BM3D cannot use the same algorithm as NSCT to achieve image enhancement. In [33], a joint image enhancement and denoising algorithm by 3D transform-domain collaborative filtering was proposed. In this algorithm, block-matching and 3D transform were implemented. Additionally, the hard thresholding operation on the 3D transformed coefficients was used to remove noise, and the alpha-rooting method was used to amplify coefficients and achieve image enhancement.Given a transform spectrumt of a signal, which contains a DC coefficient termed t(0), the alpha-rooting is performed as (6)tshi=signti∙t0∙tit01/α,ift0≠0ti,otherwisewhere tsh is the spectrum of the transformed signal and an exponent α>1 leads to the enhancement of image details.In [33], two different approaches are given to perform image enhancement: BM3D-SH3D and BM3D-SH2D. The first one implements the alpha-rooting algorithm on the 3D transformed spectrum, while the second implements the alpha-rooting algorithm only on the 2D transformed spectrum after the 1D inverse transform, i.e., on each block-transformed spectrum. However, actually, both methods achieve image enhancement only via one block in a certain sense; because of extreme similarity among the blocks in a group, the magnitudes of the coefficients in the high-frequency subbands are all nearly zero.Accordingly, to achieve better enhancement on image details, we propose using a K-neighbor block-extraction method to replace the block-matching procedure in BM3D and further discard the 2D transform on each image block by only implementing an interblock 1D Haar transform.


## 2.2.1. Image Enhancement by NSCT

NSCT has excellent linear singularity representation performance, which allows some relatively weak edges in images to be better represented by NSCT than by BM3D. Therefore, when using NSCT to achieve better image enhancement, strong edges and weak edges should be processed differently. In [7], to achieve better image enhancement results, the definition below is used to differentiate among strong edge, weak edge, and noise: (4)strongedge,ifmean≥cσweakedge,ifmean<cσ,max≥cσnoise,ifmean<cσ,max<cσwhere c is a parameter ranging from 1 to 5 and σ is the standard deviation of noise in the subbands at a specific pyramidal level. In this case, a NSCT-based image enhancement algorithm can be given as follows:(5)yx=x,strongedgepixelsmax⁡cσxp,1x,weakedgepixels0,noisepixelswhere the input x is the original transform coefficient and 0<p<1 is the amplifying gain. This function keeps the coefficients of strong edges, amplifies the coefficients of weak edges, and zeros the noise coefficients.


## 2.2.2. Image Enhancement by BM3D

According to the analysis in Section2.1, BM3D cannot obtain the linear singularity representation like NSCT. Therefore, BM3D cannot use the same algorithm as NSCT to achieve image enhancement. In [33], a joint image enhancement and denoising algorithm by 3D transform-domain collaborative filtering was proposed. In this algorithm, block-matching and 3D transform were implemented. Additionally, the hard thresholding operation on the 3D transformed coefficients was used to remove noise, and the alpha-rooting method was used to amplify coefficients and achieve image enhancement.Given a transform spectrumt of a signal, which contains a DC coefficient termed t(0), the alpha-rooting is performed as (6)tshi=signti∙t0∙tit01/α,ift0≠0ti,otherwisewhere tsh is the spectrum of the transformed signal and an exponent α>1 leads to the enhancement of image details.In [33], two different approaches are given to perform image enhancement: BM3D-SH3D and BM3D-SH2D. The first one implements the alpha-rooting algorithm on the 3D transformed spectrum, while the second implements the alpha-rooting algorithm only on the 2D transformed spectrum after the 1D inverse transform, i.e., on each block-transformed spectrum. However, actually, both methods achieve image enhancement only via one block in a certain sense; because of extreme similarity among the blocks in a group, the magnitudes of the coefficients in the high-frequency subbands are all nearly zero.Accordingly, to achieve better enhancement on image details, we propose using a K-neighbor block-extraction method to replace the block-matching procedure in BM3D and further discard the 2D transform on each image block by only implementing an interblock 1D Haar transform.


## 3. K-Neighbor Block-Extraction and Haar Transform

Considering the effectiveness of the proposed linear singularity representation method, the extracted image blocks should have similar smooth backgrounds as well as some different local details. We first select an image block from the input image and then extract its spatial K-neighbor blocks. Thus, the extracted blocks can follow the above-mentioned condition; i.e., all the extracted image blocks have similar smooth backgrounds but different local details. After a fast Haar transform on the group of these image blocks, the image details can be effectively represented in the details subbands.


### 3.1. K-Neighbor Block-Extraction

A reference image blockBR with the top-left pixel coordinate xR∈X (where X⊂Z2 is the coordinate set of the input image I) is first selected by an n×n sliding window according to a given sliding step size nstep, and then its spatial K-neighbor image blocks are extracted to form a vector BE={B1,⋯,Bi,⋯,BK}T, where every block Bi can be considered as an element of BE, and their top-left pixel coordinates xi form a set SxR={xi∈X:BiisextractedaccordingtoBR}.Because we want to implement a Haar transform on vectorBE, K must be a power-of-two integer. In addition, to represent more directional details, K should at least be 8. For example, we can extract 8 image blocks and then form all of their top-left pixel coordinates as the 8-neighborhood top-left pixel coordinates of BR. A block-extraction operation is illustrated in Figure 4. The pink block is an 8×8 reference block BR, with the blue solid round as its top-left pixel coordinate. The green solid rounds are the top-left pixel coordinates of the extracted K=8 image blocks, while the red solid rounds are the top-left pixel coordinates of the extracted K=16 image blocks.Figure 4
An illustration of the image block-extraction method.


### 3.2. Haar Transform

In this section, we give a fast Haar transform method according to the characteristics of block-extraction. Because all extracted blocks are isometric, we can fully investigate the simplicity of Haar wavelet to construct a fast Haar transform on each group of blocks.Forward Transform. Here, K=2j(j=3,4,⋯) image blocks are denoted as Bi, i=1,2,3,⋯,K. For example, if K=8, we can use the following formulation to realize a complete Haar transform with 3 levels of transform:(7)B^i=∑j=1,⋯,KΨi,jBj,i=1,⋯,Kwhere Ψ is a Haar transform matrix, BE={B1,B2,B3,⋯,BK}T is a column vector in which every element denotes an image block, and B^E={B^1,B^2,B^3,⋯,B^K}T is also a column vector in which every element denotes a transformed subband. An 8×8Ψ of Haar transform can be defined as follows:(8)Ψ=181818181818181818181818-18-18-18-181414-14-14000000001414-14-1412-120000000012-120000000012-120000000012-12By computing the matrix product, BE can be decomposed into 8 subbands, with B^1 as the approximated subband and the rest as detailed subbands. Figure 5 shows a way of decomposing a group of image blocks with a Haar transform. Note that the contours can be effectively represented in the detailed subbands.Figure 5
Example of decomposing a group of 8 image blocks (upper row) with a Haar transform. The upper row shows the 8 extracted blocks, and the lower row shows Haar-transformed subbands.Inverse Transform. The Haar transform matrix defined in (2) is invertible. Thus, one can perfectly reconstruct all original image blocks using the following inverse transform:(9)B^^i=∑j=1,⋯,KΨ-1i,jB^j,i=1,⋯,Kwhere Ψ-1 denotes the inverse matrix of Ψ.After finishing operations on a group of image blocks, we can return them to their original locations in a zero matrix of sizeM×N by averaging all pixels in the same location. By finishing operations on all reference blocks, we can get the output image I^ by the following aggregation equation:(10)I^=∑xR∈X∑xi∈SxRB^^i∑xR∈X∑xi∈SxRχiwhere χi is the characteristic function of the square support of a block located at xi, and all of the image blocks are outside-padded by zeros to form an M×N image. Because the Haar transform is perfect in reconstruction transform, we always calculate the average of all pixels extracted in the same location in the aggregation procedure. Namely, the pixel value always returns to the original value at every location, so a perfect reconstructed image can be obtained after conducting all of the above operations. Figure 6 shows a Lena image and its reconstructed image with PSNR of 313.78.Figure 6
A perfect reconstruction example of the proposed K-neighbor block-extraction and Haar (KN-BEH) transform. The left is the input image, and the right is the reconstructed image (the parameters for KN-BEH arenstep=3, n=8, K=8.).


## 3.1. K-Neighbor Block-Extraction

A reference image blockBR with the top-left pixel coordinate xR∈X (where X⊂Z2 is the coordinate set of the input image I) is first selected by an n×n sliding window according to a given sliding step size nstep, and then its spatial K-neighbor image blocks are extracted to form a vector BE={B1,⋯,Bi,⋯,BK}T, where every block Bi can be considered as an element of BE, and their top-left pixel coordinates xi form a set SxR={xi∈X:BiisextractedaccordingtoBR}.Because we want to implement a Haar transform on vectorBE, K must be a power-of-two integer. In addition, to represent more directional details, K should at least be 8. For example, we can extract 8 image blocks and then form all of their top-left pixel coordinates as the 8-neighborhood top-left pixel coordinates of BR. A block-extraction operation is illustrated in Figure 4. The pink block is an 8×8 reference block BR, with the blue solid round as its top-left pixel coordinate. The green solid rounds are the top-left pixel coordinates of the extracted K=8 image blocks, while the red solid rounds are the top-left pixel coordinates of the extracted K=16 image blocks.Figure 4
An illustration of the image block-extraction method.


## 3.2. Haar Transform

In this section, we give a fast Haar transform method according to the characteristics of block-extraction. Because all extracted blocks are isometric, we can fully investigate the simplicity of Haar wavelet to construct a fast Haar transform on each group of blocks.Forward Transform. Here, K=2j(j=3,4,⋯) image blocks are denoted as Bi, i=1,2,3,⋯,K. For example, if K=8, we can use the following formulation to realize a complete Haar transform with 3 levels of transform:(7)B^i=∑j=1,⋯,KΨi,jBj,i=1,⋯,Kwhere Ψ is a Haar transform matrix, BE={B1,B2,B3,⋯,BK}T is a column vector in which every element denotes an image block, and B^E={B^1,B^2,B^3,⋯,B^K}T is also a column vector in which every element denotes a transformed subband. An 8×8Ψ of Haar transform can be defined as follows:(8)Ψ=181818181818181818181818-18-18-18-181414-14-14000000001414-14-1412-120000000012-120000000012-120000000012-12By computing the matrix product, BE can be decomposed into 8 subbands, with B^1 as the approximated subband and the rest as detailed subbands. Figure 5 shows a way of decomposing a group of image blocks with a Haar transform. Note that the contours can be effectively represented in the detailed subbands.Figure 5
Example of decomposing a group of 8 image blocks (upper row) with a Haar transform. The upper row shows the 8 extracted blocks, and the lower row shows Haar-transformed subbands.Inverse Transform. The Haar transform matrix defined in (2) is invertible. Thus, one can perfectly reconstruct all original image blocks using the following inverse transform:(9)B^^i=∑j=1,⋯,KΨ-1i,jB^j,i=1,⋯,Kwhere Ψ-1 denotes the inverse matrix of Ψ.After finishing operations on a group of image blocks, we can return them to their original locations in a zero matrix of sizeM×N by averaging all pixels in the same location. By finishing operations on all reference blocks, we can get the output image I^ by the following aggregation equation:(10)I^=∑xR∈X∑xi∈SxRB^^i∑xR∈X∑xi∈SxRχiwhere χi is the characteristic function of the square support of a block located at xi, and all of the image blocks are outside-padded by zeros to form an M×N image. Because the Haar transform is perfect in reconstruction transform, we always calculate the average of all pixels extracted in the same location in the aggregation procedure. Namely, the pixel value always returns to the original value at every location, so a perfect reconstructed image can be obtained after conducting all of the above operations. Figure 6 shows a Lena image and its reconstructed image with PSNR of 313.78.Figure 6
A perfect reconstruction example of the proposed K-neighbor block-extraction and Haar (KN-BEH) transform. The left is the input image, and the right is the reconstructed image (the parameters for KN-BEH arenstep=3, n=8, K=8.).


## 4. Application to Image Enhancement

We apply the proposed linear singularity representation method to image enhancement to verify its effectiveness in capturing image details among a group of blocks. Amplifying certain transformed coefficients can typically achieve image enhancement, i.e., amplifying only the transformation coefficients of image details but suppressing transformation coefficients of background noise [7, 34, 35]. If image details cannot be well represented, the transformation coefficients of background noises may also be amplified when amplifying transformation coefficients. In addition, when implementing the traditional convolution-based transform, image details and their surrounding smooth background can be influenced by each other because the transformation coefficients of the background near the image details are usually larger than those that are far away. Thus, the halo phenomenon would be introduced after image enhancement. However, due to the effectiveness of our proposed linear singularity representation, it can amplify the transformation coefficients of image details and also suppress the transformation coefficients of background and noise simultaneously. Most importantly, without using large filter banks, the halo phenomenon can be effectively alleviated.For the purpose of suppressing background noise, we first estimate the noise deviationσ of the input image with the robust median operator [36].As in the analysis in Section3.2, we implement the forward Haar transform on BE={B1,B2,B3,⋯,BK}T according to (7) to obtain the transformation coefficients B^E={B^1,B^2,B^3,⋯,B^K}T. Then, we can use the following nonlinear mapping function to amplify the transformation coefficients of image details,(11)Cen=Ct,ifCt>CσγCt,ifcσ≤Ct≤Cσ0,ifCt<cσwhere Ct is the Haar-transformed coefficients, γ is the gain factor to amplify the transformation coefficients of image details, C and c are the two constant parameters, and Cen is the enhanced coefficient. Next, we can implement the inverse Haar transform to get B^^E=(B^^1,B^^2,B^^3,⋯B^^K)T by (9). Finally, we can use (10) to obtain the enhanced image.The image enhancement algorithm by BEH can be summarized as follows:(1)
Estimate the noise deviation of the input image.(2)
ExtractK image blocks by the method in Section 2.1.(3)
Implement Haar transform according to (7).(4)
Amplify the transformation coefficients of image details and also suppress the transformation coefficients of background noise by (11).(5)
Implement the inverse Haar transform according to (9).(6)
Aggregate all of the blocks to obtain the final enhanced image.The proposed BEH method has two drawbacks for image denoising:(1) the extracted image blocks are not similar enough to each other, and (2) this algorithm does not implement the 2D transform on each block. These two drawbacks limit separation of noise from signal. Fortunately, because the BM3D method can achieve outstanding denoising performance, we can use BM3D to denoise a noisy image and then use the proposed method to enhance the details of the denoised image. This overall method is called BEH-BM3D.It is worth noting that there are two steps in BM3D method for achieving better image denoising. The first step can basically remove all the noise in the noisy image; however the image details are also oversmoothed. In order to restore the oversmoothed image details, BM3D uses the second step, i.e., Wiener filtering. This two-step strategy is needed for image denoising problem; but, for image enhancement, the Wiener filtering step is not suitable, since the enhanced image would be restored to the original one if applying Wiener filtering on the enhanced image. We only implement one-step block-extraction Haar transform in our method; thus the computational complex can also be lowered than the BM3D method.


## 5. Experimental Results

In this section, we will provide two groups of experiments. One is to implement the proposed linear singularity representation method, and the other is to implement image enhancement.


### 5.1. Linear Singularity Representation

To demonstrate the proposed linear singularity representation method, the Barbara image with a size of512×512 is decomposed via the proposed KN-BEH. The parameter values used are nstep=3, n=8, and K=8. Figure 7 shows all the transformed subbands of the proposed method. Obviously, image details can be observed in the detailed subbands. This is mainly because the KN-BEH can easily capture differences among the blocks of the same group, where neighboring image blocks have very similar backgrounds.Figure 7
The transformed subbands of the proposed transform. To clearly display the details, we have increased the brightness of each high-frequency subband.


### 5.2. Image Enhancement

We have conducted image enhancement experiments to verify the effectiveness of our proposed method. Whenγ>1, the output image details will be enhanced, with the larger γ correlated to a more strongly enhanced image. In all experiments, the parameters are set as C=30.0 and c=8.0 for the normalized input image, i.e., with the pixel value ranging from 0 to 1. We just adjust γ to get different image enhancement results. The enhancement results on House and Barbara are shown in Figures 8 and 9.Figure 8
The enhancement results. (a) is the original image. (b) and (c) are the enhanced images by the proposed method, by takingγ as 3 and 5, respectively. (d) and (e) are the enhanced images by the NSCT method, by similarly taking γ as 3 and 5, respectively. (f) is the enhanced image by the alpha-rooting method.
(a) (b) (c) (d) (e) (f)Figure 9
Comparison of image enhancement results by the proposed method, NSCT, and alpha-rooting. (a) Input image, (b) the enhanced image by the proposed method, (c) the enhanced image by NSCT, and (d) the enhanced image by alpha-rooting.
(a) (b) (c) (d)To further demonstrate the advantage of the BEH method, we also compare our image enhancement results with results using the NSCT method [7] and the alpha-rooting method [33]. Due to the use of multidirectional filtering banks and translation invariants, NSCT is a powerful linear singularity representation method. Figures 8 and 9 show the comparison of image enhancement results by our proposed method, NSCT, and alpha-rooting. We observe that the proposed method hardly produces any halo phenomenon around strong edges, but the NSCT method results in a strong halo phenomenon. From Figures 8 and 9, we also observe that our proposed method can better preserve weak details than the NSCT method, i.e., for the details on the table legs at the bottom-left part of the Barbara image. Because our method can better represent linear singularity, as observed from Figures 8 and 9, our proposed method has better subjective visual sharpness than the alpha-rooting method. Additionally, many small, weak details in the image cannot be sharpened using the alpha-rooting method.We have further conducted other comparison experiments. For example, we used both BEH-BM3D and alpha-rooting BM3D to enhance images with added Gaussian white noise. Specifically, we first added different levels of noise to the image and then used these two methods to enhance the image and remove noise. For BEH-BM3D, we used a two-stage method, with the first stage of image denoising by BM3D and the second stage of enhancement by BEH. For the alpha-rooting BM3D method, by exactly following the procedure in [33], we simultaneously enhanced the transformation coefficients of image details and removed the transformation coefficients of noise by hard thresholding in BM3D transform domain. The experimental results are shown in Figures 10 and 11. From these two figures, we can see that the BEH-BM3D can better preserve image details and achieve better subjective visual results.Figure 10
Image enhancement results by the alpha-rooting BM3D. The first row is the original House image, noisy image withσ=10, and noisy image with σ=20, respectively, from left to right. Row 2 to row 6 give the corresponding simultaneous denoising and enhancement results by using α=1.2,1.4,1.6,1.8,2.0, respectively.Figure 11
Image enhancement results by BEH-BM3D using different gain factorsγ=2,3,5,8,10 from top row to bottom row, respectively. The left column shows the enhancement results of the original image, the middle column shows the enhancement results of image with added noise σ=10, and the right column shows the enhancement results of image with added noise σ=20.To quantitatively validate image enhancement performance, the Local Phase Coherence Sharpness Index (LPC-SI) [37], a novel and effective image sharpness assessment method, is used. This method considers sharpness as strong local phase coherence near the distinctive image features evaluated in the complex wavelet transform domain. Furthermore, this method can assess sharpness without using the original image as reference. In our experiments, we compare our results with both the NSCT method and the alpha-rooting method. In each method, we provide its best LPC-SI results by adjusting the respective parameters. For the alpha-rooting method, we fix the alpha value to 2.0 and then adjust the standard deviation of noise to obtain the best LPC-SI values. For the NSCT method, we use the enhancement algorithm in [7] by adjusting the gain factor c in (5) to obtain the best LPC-SI values. All of the results are shown in Table 1. We can see that our method achieves the best results for most images.Table 1
The comparison of LPC-SI values in the original images and the enhanced images by the NSCT method, Alpha-rooting method, and our proposed method. All of the values are the best ones for the respective method.
Image LPC-SI LPC-SI LPC-SI LPC-SI input NSCT alpha-rooting our proposed Barbara 0.9246 0.9532 0.9685 0.9682  House 0.8760 0.9302 0.9395 0.9603  Boat 0.9341 0.9503 0.9529 0.9590  Lena 0.9060 0.9411 0.9549 0.9629  Peppers 0.9210 0.9533 0.9473 0.9592  Couple 0.9213 0.9535 0.9499 0.9565  Hill 0.8840 0.9438 0.9422 0.9566  Man 0.9104 0.9508 0.9499 0.9594  Cameraman 0.9364 0.9578 0.9534 0.9559  Fingerprint 0.7777 0.9401 0.9034 0.9204In addition, we further used the traditional background variation (BV) and detail variation (DV) [35] to evaluate our results. The BV and DV values represent the variance of background and foreground pixels, respectively. A good image enhancement method should increase the DV of the original image but keep or even decrease the BV. The BV and DV values of three methods are summarized in Table 2. Notice that all of these results were obtained for each image and each method when the corresponding LPC-SI was the best. From these results, we can see that the alpha-rooting method achieved the lowest BV, and our method achieved the highest DV. However, the BV values provided by our method were just a little bit higher than those of the alpha-rooting method. If we use the ratio between DV and BV to assess the obtained results, our method achieves the best outcome.Table 2
The BV and DV values in the original image and the enhanced images by the NSCT method, Alpha-root method, and our proposed method. All of these values are obtained when the LPC-SI is the best.
Image Input Alpha-rooting NSCT Our proposed BV DV BV DV BV DV BV DV Lena 0.55 6.60 0.34 13.38 0.40 16.85 0.36 18.38  House 0.62 7.05 0.12 24.14 0.26 20.77 0.12 32.33  Barbara 0.66 12.53 0.33 32.57 0.48 31.48 0.37 33.50  Peppers 0.68 10.43 0.31 19.74 0.47 13.04 0.42 31.02  Boat 0.66 9.47 0.33 15.43 0.47 24.40 0.34 23.07  Couple 0.63 9.95 0.34 24.20 0.45 25.65 0.40 30.47  Hill 0.58 8.42 0.30 18.52 0.41 18.78 0.34 21.81  Man 0.53 8.60 0.28 19.67 0.48 21.66 0.34 29.22  Cam. 0.60 11.01 0.26 28.36 0.36 35.62 0.35 43.79  Fingerprint 0.53 18.95 0.39 61.97 0.57 10.47 0.52 50.54We also find that NSCT achieves the best LPC-SI value, but the worst BV/DV result on Fingerprint images in Tables1 and 2, respectively. This is mainly because the estimated standard deviation of noise is too high, which results in most image edges being removed as noise. Therefore, the highest LPC-SI value cannot always be trusted. This is also the main reason for us to use both LPC-SI and BV/DV to assess the experimental results.It is worth indicating another significant advantage of our proposed method, i.e., its lower time complexity compared to that of the NSCT method. For image enhancement on a256×256 grayscale image, our method takes only 0.3 second, while the NSCT method takes 12.6 seconds. Our method uses the same parameter values, i.e., nstep=3, n=8, and K=8, and the NSCT method uses 2 levels of decomposition, i.e., 2 and 4 directional subbands, respectively.In the final experiment, we also used the proposed BEH method to enhance several color images to show superior enhancement performance and compare the results with the guided image filtering method [20, 21], with the results shown in Figure 12. Although the image details are enhanced, the enhanced images are still visually natural when using the proposed method. The guided image filtering method can also effectively enhance the image details; however, the image background is also enhanced, making the enhanced images look rather artificial.Figure 12
Enhanced color images. Left are the original images, middle are the enhanced images by guided image filtering, and right are the enhanced images by BEH.


## 5.1. Linear Singularity Representation

To demonstrate the proposed linear singularity representation method, the Barbara image with a size of512×512 is decomposed via the proposed KN-BEH. The parameter values used are nstep=3, n=8, and K=8. Figure 7 shows all the transformed subbands of the proposed method. Obviously, image details can be observed in the detailed subbands. This is mainly because the KN-BEH can easily capture differences among the blocks of the same group, where neighboring image blocks have very similar backgrounds.Figure 7
The transformed subbands of the proposed transform. To clearly display the details, we have increased the brightness of each high-frequency subband.


## 5.2. Image Enhancement

We have conducted image enhancement experiments to verify the effectiveness of our proposed method. Whenγ>1, the output image details will be enhanced, with the larger γ correlated to a more strongly enhanced image. In all experiments, the parameters are set as C=30.0 and c=8.0 for the normalized input image, i.e., with the pixel value ranging from 0 to 1. We just adjust γ to get different image enhancement results. The enhancement results on House and Barbara are shown in Figures 8 and 9.Figure 8
The enhancement results. (a) is the original image. (b) and (c) are the enhanced images by the proposed method, by takingγ as 3 and 5, respectively. (d) and (e) are the enhanced images by the NSCT method, by similarly taking γ as 3 and 5, respectively. (f) is the enhanced image by the alpha-rooting method.
(a) (b) (c) (d) (e) (f)Figure 9
Comparison of image enhancement results by the proposed method, NSCT, and alpha-rooting. (a) Input image, (b) the enhanced image by the proposed method, (c) the enhanced image by NSCT, and (d) the enhanced image by alpha-rooting.
(a) (b) (c) (d)To further demonstrate the advantage of the BEH method, we also compare our image enhancement results with results using the NSCT method [7] and the alpha-rooting method [33]. Due to the use of multidirectional filtering banks and translation invariants, NSCT is a powerful linear singularity representation method. Figures 8 and 9 show the comparison of image enhancement results by our proposed method, NSCT, and alpha-rooting. We observe that the proposed method hardly produces any halo phenomenon around strong edges, but the NSCT method results in a strong halo phenomenon. From Figures 8 and 9, we also observe that our proposed method can better preserve weak details than the NSCT method, i.e., for the details on the table legs at the bottom-left part of the Barbara image. Because our method can better represent linear singularity, as observed from Figures 8 and 9, our proposed method has better subjective visual sharpness than the alpha-rooting method. Additionally, many small, weak details in the image cannot be sharpened using the alpha-rooting method.We have further conducted other comparison experiments. For example, we used both BEH-BM3D and alpha-rooting BM3D to enhance images with added Gaussian white noise. Specifically, we first added different levels of noise to the image and then used these two methods to enhance the image and remove noise. For BEH-BM3D, we used a two-stage method, with the first stage of image denoising by BM3D and the second stage of enhancement by BEH. For the alpha-rooting BM3D method, by exactly following the procedure in [33], we simultaneously enhanced the transformation coefficients of image details and removed the transformation coefficients of noise by hard thresholding in BM3D transform domain. The experimental results are shown in Figures 10 and 11. From these two figures, we can see that the BEH-BM3D can better preserve image details and achieve better subjective visual results.Figure 10
Image enhancement results by the alpha-rooting BM3D. The first row is the original House image, noisy image withσ=10, and noisy image with σ=20, respectively, from left to right. Row 2 to row 6 give the corresponding simultaneous denoising and enhancement results by using α=1.2,1.4,1.6,1.8,2.0, respectively.Figure 11
Image enhancement results by BEH-BM3D using different gain factorsγ=2,3,5,8,10 from top row to bottom row, respectively. The left column shows the enhancement results of the original image, the middle column shows the enhancement results of image with added noise σ=10, and the right column shows the enhancement results of image with added noise σ=20.To quantitatively validate image enhancement performance, the Local Phase Coherence Sharpness Index (LPC-SI) [37], a novel and effective image sharpness assessment method, is used. This method considers sharpness as strong local phase coherence near the distinctive image features evaluated in the complex wavelet transform domain. Furthermore, this method can assess sharpness without using the original image as reference. In our experiments, we compare our results with both the NSCT method and the alpha-rooting method. In each method, we provide its best LPC-SI results by adjusting the respective parameters. For the alpha-rooting method, we fix the alpha value to 2.0 and then adjust the standard deviation of noise to obtain the best LPC-SI values. For the NSCT method, we use the enhancement algorithm in [7] by adjusting the gain factor c in (5) to obtain the best LPC-SI values. All of the results are shown in Table 1. We can see that our method achieves the best results for most images.Table 1
The comparison of LPC-SI values in the original images and the enhanced images by the NSCT method, Alpha-rooting method, and our proposed method. All of the values are the best ones for the respective method.
Image LPC-SI LPC-SI LPC-SI LPC-SI input NSCT alpha-rooting our proposed Barbara 0.9246 0.9532 0.9685 0.9682  House 0.8760 0.9302 0.9395 0.9603  Boat 0.9341 0.9503 0.9529 0.9590  Lena 0.9060 0.9411 0.9549 0.9629  Peppers 0.9210 0.9533 0.9473 0.9592  Couple 0.9213 0.9535 0.9499 0.9565  Hill 0.8840 0.9438 0.9422 0.9566  Man 0.9104 0.9508 0.9499 0.9594  Cameraman 0.9364 0.9578 0.9534 0.9559  Fingerprint 0.7777 0.9401 0.9034 0.9204In addition, we further used the traditional background variation (BV) and detail variation (DV) [35] to evaluate our results. The BV and DV values represent the variance of background and foreground pixels, respectively. A good image enhancement method should increase the DV of the original image but keep or even decrease the BV. The BV and DV values of three methods are summarized in Table 2. Notice that all of these results were obtained for each image and each method when the corresponding LPC-SI was the best. From these results, we can see that the alpha-rooting method achieved the lowest BV, and our method achieved the highest DV. However, the BV values provided by our method were just a little bit higher than those of the alpha-rooting method. If we use the ratio between DV and BV to assess the obtained results, our method achieves the best outcome.Table 2
The BV and DV values in the original image and the enhanced images by the NSCT method, Alpha-root method, and our proposed method. All of these values are obtained when the LPC-SI is the best.
Image Input Alpha-rooting NSCT Our proposed BV DV BV DV BV DV BV DV Lena 0.55 6.60 0.34 13.38 0.40 16.85 0.36 18.38  House 0.62 7.05 0.12 24.14 0.26 20.77 0.12 32.33  Barbara 0.66 12.53 0.33 32.57 0.48 31.48 0.37 33.50  Peppers 0.68 10.43 0.31 19.74 0.47 13.04 0.42 31.02  Boat 0.66 9.47 0.33 15.43 0.47 24.40 0.34 23.07  Couple 0.63 9.95 0.34 24.20 0.45 25.65 0.40 30.47  Hill 0.58 8.42 0.30 18.52 0.41 18.78 0.34 21.81  Man 0.53 8.60 0.28 19.67 0.48 21.66 0.34 29.22  Cam. 0.60 11.01 0.26 28.36 0.36 35.62 0.35 43.79  Fingerprint 0.53 18.95 0.39 61.97 0.57 10.47 0.52 50.54We also find that NSCT achieves the best LPC-SI value, but the worst BV/DV result on Fingerprint images in Tables1 and 2, respectively. This is mainly because the estimated standard deviation of noise is too high, which results in most image edges being removed as noise. Therefore, the highest LPC-SI value cannot always be trusted. This is also the main reason for us to use both LPC-SI and BV/DV to assess the experimental results.It is worth indicating another significant advantage of our proposed method, i.e., its lower time complexity compared to that of the NSCT method. For image enhancement on a256×256 grayscale image, our method takes only 0.3 second, while the NSCT method takes 12.6 seconds. Our method uses the same parameter values, i.e., nstep=3, n=8, and K=8, and the NSCT method uses 2 levels of decomposition, i.e., 2 and 4 directional subbands, respectively.In the final experiment, we also used the proposed BEH method to enhance several color images to show superior enhancement performance and compare the results with the guided image filtering method [20, 21], with the results shown in Figure 12. Although the image details are enhanced, the enhanced images are still visually natural when using the proposed method. The guided image filtering method can also effectively enhance the image details; however, the image background is also enhanced, making the enhanced images look rather artificial.Figure 12
Enhanced color images. Left are the original images, middle are the enhanced images by guided image filtering, and right are the enhanced images by BEH.


## 6. Conclusions

We have proposed an effective linear singularity representation method and have applied this method to image enhancement in both gray and color images. Similar to the BM3D method, our method is nonlocal. However, our method extracts some spatially adjacent blocks, instead of using block-matching. By using our method, the image details can be effectively represented. Additionally, by using different parameters to amplify the transformation coefficients of image details, we can obtain different image enhancement results. Because our method uses no local convolution operation, it introduces fewer halo phenomena, compared to the NSCT method. Furthermore, the computational cost of our method is also significantly lower, compared to that of the NSCT method.Although our proposed BEH method looks like the original BM3D method, the purpose of our proposed method is to represent image details, which is different from the original BM3D. Note that edges in the matched image blocks are usually located in the same position; thus, edge information cannot be preserved in the high-frequency subbands after the third-dimensional transform. Therefore, our proposed method is different from the BM3D method.In summary, we have presented a simple but effective image linear singularity representation method that achieves better objective and subjective results when applied to image enhancement than other state-of-the-art methods.



---

*Source: 6395147-2019-08-06.xml*