# Image Noise Removal Method Based on Improved Nonlocal Mean Algorithm

Authors: Keya Huang; Hairong Zhu
Journal: Complexity
 (2021)

---

## Abstract

Aiming at the problem of unclear images acquired in interactive systems, an improved image processing algorithm for nonlocal mean denoising is proposed. This algorithm combines the adaptive median filter algorithm with the traditional nonlocal mean algorithm, first adjusts the image window adaptively, selects the corresponding pixel weight, and then denoises the image, which can have a good filtering effect on the mixed noise. The experimental results show that, compared with the traditional nonlocal mean algorithm, the algorithm proposed in this paper has better results in the visual quality and peak signal-to-noise ratio (PSNR) of complex noise images.


---

## Body

## 1. Introduction

The data signal in the interactive system will be interfered by noise during the collection and transmission process. Random noise reduces image quality and conceals detailed feature information, which is not conducive to further analysis of feature recognition and target tracking. To improve the image quality of the interactive system and ensure the accuracy of the interaction, the current denoising algorithm needs to be optimized [1, 2]. Hassouni et al. removed image noise based on the HOS sequence. The motion trajectories are obtained recursively by a region-recursive estimation method [3]. Many scholars have designed detection methods for extracting image edges from noisy images [4, 5]. At present, in the image denoising problem of interactive systems, most of the research is to denoise a single type of noise. However, mixed Gaussian and salt and pepper noises are common in interactive environments, so it is of great practical significance to study the problem of mixed noise filtering [6–8].Denoising methods are mainly divided into three areas: spatial domain denoising, transform domain denoising, and statistical method denoising. Spatial domain noise reduction refers to the use of spatial filtering algorithms for noise reduction processing. This algorithm acts directly on the observable pixels and depends on the target pixel and neighboring pixels. Spatial filters include linear filters [9] and nonlinear filters. Common linear filtering algorithms include Wiener filtering and median filtering [10]. The disadvantage of Wiener filtering is that the image data must be smooth, so its application is limited. The median filter replaces the intermediate gray value of the target pixel with the gray value of the pixel in a certain window, which is particularly effective for removing salt and pepper noise [11]. The mean filter is a common type of nonlinear filter, which replaces the target pixel intensity with the average pixel intensity of the target neighborhood in a specific window. There is also a bilateral mesh denoising algorithm that can effectively filter the image [12]. The above algorithms are simple and can effectively remove the corresponding noise. However, due to the different mathematical characteristics of different noises, it is difficult for these algorithms to be effective for multiple noises at the same time, and they will all cause the edges of the image to become blurred to a certain extent. As the window increases, the image will become more blurred [13].Nonlocal mean (NLM) algorithm is an excellent spatial denoising algorithm [14–16]. The algorithm uses the characteristics of many similar blocks in the image and the average value of similar blocks to suppress image noise. At the beginning, Buades et al. [17] proposed a nonlocal average denoising algorithm based on the similarity (periodicity) of image features. Before this, the periodicity of the image was mainly used for texture synthesis. In recent years, there has been a lot of research and improvement on NLM algorithm denoising methods, which are mainly used for compressed sensing, feature extraction, pattern recognition, and image denoising. With the development of mathematical theory and image processing technology, good mathematical characteristics make its denoising advantages more obvious, and this algorithm has become one of the well-known denoising algorithms. There are also many scholars who have improved the classic NLM algorithm, especially the steps of dividing the pixel area of the traditional algorithm, so that it can retain more image structure information [18]. For example, the patch-based NLM algorithm is used in the field of medical image recognition to improve the fidelity of the image [19]. Although the principle of NLM algorithm is simple and the denoising effect is obvious, the calculation is not simple. The main reason is that the calculation of the similarity weight is complicated and takes a lot of calculation time. Therefore, how to remove pixels with little correlation while maintaining the noise reduction effect to reduce the amount of calculation will bring practical performance improvements to the NLM algorithm. The NLM algorithm has a weak filtering ability for salt and pepper noise. Therefore, it is necessary to improve the NLM algorithm to make it equally effective for salt and pepper noise, thereby enhancing the denoising performance and improving the image quality of the interactive system.Aiming at some of the shortcomings of the traditional NLM algorithm, this article conducts in-depth research on it and introduces an adaptive median filtering (AMF) algorithm, and an improved NLM algorithm for the processing of mixed color noise is proposed, which is used to better denoise the image information after information fusion.


## 2. Denoising Algorithm Based on NLM Algorithm

In the process of weighted enhancement of pixels using the NLM algorithm, similar pixels are selected in a considerable range (which can be the entire image) instead of just taking a small neighborhood like traditional spatial filtering. The NLM algorithm was originally proposed for Gaussian noise, and the Gaussian noise model is(1)Y=X+N.Y represents the image to be processed, and X represents the original clear image. Ω=1,…m×1,…,n is the range of pixel coordinates of the image. N∈l2 (Ω) is the image noise. Assume that X^ represents the denoising result for any pixel coordinate i=i1,i2∈Ω. The NLM filtering result of the gray value Xi of the pixel is the weighted average of the gray values of all pixels in the noise-containing image Y. The corresponding mathematical model is as follows:(2)X^i=∑j∈ΩwijYj=1Ci∑j∈Ωexp−Yi−Yj2,α2h2Yj.Yi=YNdi=Yj|j∈Ndi represents the image block centered on the pixel point i, the filter coefficient is set to 10σ, and Ci represents the Gaussian weighted dissimilarity of the image block and the pixel. Its specific definition is as follows:(3)Yi−Yj2,α2=∑GαkYi−k−Yj−k2.K=k1,k2k1≤d,k2≤d is the neighborhood of the pixel coordinates with the origin as the center point and the radius d, and(4)Gαk=12πα2exp−k12+k222α2,k=k1,k2,where Gαk is a Gaussian kernel with a standard deviation of α. Equation (3) also satisfies(5)EYi−Yj2,α2=Xi−Xj2,α2+2σ2.Compared with other traditional spatial filtering, the NLM algorithm uses the similar periodicity of nonlocal images and does not have strict requirements on image characteristics. It can be seen from equation (2) that the range of its calculation search box is an entire image in the defined interval Ω, and the 21∗21 rectangular window has the highest efficiency after testing. From the measure of dissimilarity in equation (3), the NLM algorithm calculates the weight through similar regions of the image instead of a single pixel. It has been tested that 7∗7 or 9∗9 rectangular windows have the highest efficiency.The formula of NLM algorithm to remove noise is(6)X^i=∑j∈ΩwijYj=1Ci∑j∈Ωexp−Yi−Yj2,α2h2Yj,where h is the filter parameter and Yi, Yj is the image block centered on i, j. Ci=∑j∈Ωexp−Yi−Yj2,α2/h2 is the normalized factor, Yi−Yj2,α2=∑k∈KGαkYi−k−Yj−k2 is the measure of dissimilarity between similar blocks of the image, where K=k1,k2 is the pixel neighborhood with d as the radius and the origin as the center, and Gαk is the Gaussian kernel with standard deviation α.Convert the digital image collected by the sensor into a grayscale image. LetX represent the original image without noise and Y represent the image containing noise. Ω=1,…,m×1,…,n represents the value range of pixel coordinates, and the value range of image gray is [lmin, lmax].N is Gaussian noise whose mean is 0 and the standard deviation is σ. Then, the image model with Gaussian noise is(7)Y=X+N.The transmission of interactive images generally contains salt and pepper noise. In the salt and pepper noise model, for∀i=i1,i2∈Ω,(8)Yi=lmin,p2,lmax,p2,Xi,1−p,where p is the intensity of salt and pepper noise.The above is the definition of Gaussian noise and impulse noise. The mixed noise model of the two noises is defined as(9)Y=NimpX+N.Through research, it is found that the median filter in the spatial domain, especially the AMF, has an outstanding effect on removing salt and pepper noise. Median filtering is generally divided into one-dimensional median filtering and two-dimensional median filtering. The algorithm principle of one-dimensional median filtering is to first sort the gray values of the pixels in the neighborhood of the target pixel from large to small and then take the median value from the ranked gray values to replace the gray value of the target pixel [20].AMF has been widely used as a classic excellent filter for filtering salt and pepper noise [21–24]. Yi represents the coordinate window of a rectangle of size L∗L, and Lmax∗Lmax is the maximum size of the rectangle window during the algorithm. The process steps of the AMF algorithm are shown in Algorithm 1.Algorithm 1:AMF algorithm flow steps.
Initialize and start(1)
Initialize window sizeL = 3.(2)
Calculate the minimum, median, and maximum of pixel gray values.(3)
If the minimum value of the gray value < the median value < the maximum value, go to step 5; otherwiseL = L + 2.(4)
If the window size is less than the maximum value of the window, go to step 2; otherwise replace the entire gray value with the median value.(5)
If the minimum gray value < gray value < maximum gray value, then the gray value can be retained in the interval; otherwise the median value is used to replace the entire gray value.From the algorithm flow chart in Figure1, it can be found that there are two main strategies of the AMF algorithm:(1)
Adaptive window adjustment strategy: it can automatically adjust the window size and automatically filter useful pixels that are not disturbed by noise in the neighborhood of the picture pixels.(2)
Selective enhancement strategies: the AMF algorithm keeps the gray value of uncontaminated pixels unchanged and only enhances the pixels that may be contaminated by noise.Figure 1
AMF algorithm flow chart.The principle and process of the NLM and AMF algorithms are analyzed above, and then the NLM filter will be improved, and the algorithm will be applied to interactive images to improve image quality.In the calculation of the NLM algorithm, the weight calculation between pixelsi and j is based on the measure of dissimilarity between their associated image blocks Yi and Yj. For Gaussian noise, the dissimilarity satisfies the following equation:(10)EYi−Yj2,α2=Xi−Xj2,α2+2σ2.However, the dissimilarity measure of equation (10) does not hold for salt and pepper noise. To solve this problem, inspired by AMF, the calculation of this dissimilarity measure can be directly established on the image space containing non-salt and pepper noise.First, define the spatial set of non-salt and pepper noise points in the image.Y is a noisy image, and i and j are any two pixels. Let Pmin,Pmax denote the range of gray values in the image block Yi and graphic block Yj; the non-salt and pepper noise point index space of these two image blocks is defined as follows:(11)Ku=k|k∈K,Yi−k∉Pmin,Pmax,Yj−k∉Pmin,Pmax,where K=k1,k2|k1≤d,k2≤d represents the coordinate neighborhood of the pixel with the radius d and the origin as the center point. UKu is the index space of non-salt and pepper noise points of the entire noisy image Y.Based on this index space, the new measure of dissimilarity betweenYi and Yj is as follows:(12)Yi−Yj2,α2=∑k∈KuGαkYi−k−Yj−k2.The dissimilarity measureYi−Yj2,α2 of the classic NLM algorithm is based on all pixels in Yi and Yj, while the new dissimilarity measure is based on the non-salt and pepper noise points of the two associated image blocks. For salt and pepper noise, there is(13)EYi−Yj2,α,u2=E∑k∈KuGαkYi−k−Yj−k2=E∑k∈KuGαkxi−k−xj−k2=Xi−Xj2,α,u2≈Xi−Xj2,α2,which is(14)EYi−Yj2,α,u2≈Xi−Xj2,α2.Equation (14) can explain that the new dissimilarity measurement method proposed above maintains a certain degree of consistency between the noisy image and the original image. This consistency is also true for Gaussian noise Yi−Yj2,α,u2≈Yi−Yj2,α2.|Ku| is the number of non-salt and pepper noise point index spaces. The larger the |Ku|, the stronger the consistency, and the smaller the |Ku|, the weaker the consistency. To obtain an accurate dissimilarity measure, it is necessary to adjust the size of the pixel neighborhood K and then select an appropriate |Ku|. Let dmin and dmax denote the minimum radius and maximum radius of the window area, and T is the allowable threshold of |Ku|. The specific algorithm steps for obtaining wij in the improved algorithm are shown in Algorithm 2.Algorithm 2:Improved NLM algorithm weight update steps.
Initialize, select any 2 pixelsi and j(1)
Set minimum radiusd = dmin(2)
Ifd ≤ dmax, calculate Ku, otherwise go to step 6(3)
Ifj ∉ Ku, go to step 6(4)
If|Ku|<T, then let d = d + 1, and then go to step 2(5)
LetWij=exp−Yi−Yj2,α,u2/h2, end(6)
LetWij=0, endTherefore, the final filtering result of the improved NLM algorithm is(15)Xi=∑j∈Ω,WijkYi.It can be seen that the calculation complexity of the NLM algorithm is mainly determined by the weight calculation method. Compared with the classic NLM algorithm, the improved NLM algorithm has the advantages that the image block size can be adjusted adaptively, the weight calculation between pixels is based on the image block measurement, and the pixel weight average is based on non-salt and pepper noise points for enhancement.


## 3. Analysis of Results

This section designs two sets of experiments. The first group compares the calibration interface diagrams with different noises in the interactive system. The second group selected a classical grayscale image and a picture collected by a camera, introduced different degrees of noise and mixed noise, and then analyzed the image quality evaluation coefficient. Finally, the analysis of experimental results proves that the improved NLM algorithm has excellent denoising performance.Experiment 1.
Open the interactive system and select the interactive point positioning map of the system calibration interface, and compare and observe the calibration effect map after using different denoising algorithms. The red line trace represents the interaction point of induction. If there is noise in the image, it will affect the recognition of gestures. This experiment is mainly used as a reference for the recognition of noisy images.
Figure2 shows the calibration interface after using different noise reduction algorithms. Figure 2(a) is the original image without any algorithm, Figure 2(b) is the positioning effect diagram after the NLM algorithm, Figure 2(c) is the positioning effect diagram after the ALM algorithm, and Figure 2(d) is the improved NLM algorithm effect diagram proposed in this paper.
It can be clearly seen from the results in Figure2 that the system sensing points in the image after adopting the denoising algorithm are more accurate. The improved NLM algorithm has the best effect, but it is still not obvious. Considering that the complexity of the interactive environment is changing, it is necessary to quantify the data. Thus, the following experiments introduce different degrees of different types of noise for comparison.Figure 2
System interface rendering. (a) Original image. (b) Effect picture after NLM algorithm. (c) Effect picture after ALM algorithm. (d) Effect picture after the improved NLM algorithm.
(a)(b)(c)(d)Experiment 2.
Figure3 is a denoising comparison experiment. Figure 3(a) is an ordinary grayscale image, and Lena in Figure 3(b) is a classic grayscale image. The quantified evaluation of image noise generally uses the peak signal-to-noise ratio (PSNR), and the formula is as follows:(16)PSNR=10log102552MSE.
Among them,MSE=1/Ω∑i∈ΩX^i−Xi2 is the mean square error. The larger the PSNR value, the better the denoising effect. Of course, there are many performance indicators that can measure the fidelity of the image, such as brightness, contrast, and structural similarity [25]. This paper mainly uses PSNR to judge the image fidelity.
In the experiment, the denoising effects of the improved NLM algorithm, NLM algorithm, and AMF algorithm are compared. The different parameter settings of different filters in the experiment are shown in Table1.
For the NLM algorithm,h2=10σ, the radius of the image block is 3 and the radius of the search window is 8. For the improved NLM algorithm, h2=10σ+300p is better, the radius of the search window is set to 8, and a, b, and T are set to 3, 5, and 14.
Next, the image will be polluted to different degrees, and then the denoising effects of each algorithm will be compared.
First, we introduce different degrees of salt and pepper noise to compare the effects of the two images. Table2 shows the denoising effect data of Lena images polluted by different salt and pepper noises. Table 3 shows the denoising effect data of ordinary human images polluted by different salt and pepper noises. Figure 4 shows the denoising effect of Lena image polluted by salt and pepper noise. Figure 5 is a denoising effect diagram of an image of an ordinary person polluted by salt and pepper noise.
It can be seen from the above Figures4 and 5 and Tables 2 and 3 that the improved NLM algorithm proposed in this paper has good denoising performance against salt and pepper noise in both images. The AMF algorithm is also very powerful in removing salt and pepper noise, which is not much different from the algorithm in this paper. In contrast, the NLM algorithm is not ideal for removing salt and pepper noise.
Then, introduce different degrees of Gaussian noise and then compare the experiments.
From the above Figures6 and 7 and Table 4, it can be concluded that the improved NLM algorithm has excellent filtering effect on Gaussian noise, and PSNR value and visual effect are all better than other algorithms. Although the RSNR of the proposed algorithm is slightly lower than that of the traditional NLM algorithm, the difference is not big, and the final filtering effect is not much difference.
Finally, the mixed noise is introduced into the image, and then the performance of the denoising algorithm is considered from the aspects of PSNR and visual effect. The results are as follows.
From the performance of each denoising algorithm in the above Figures8 and 9 and Tables 5 and 6, the improved NLM algorithm proposed in this paper has obvious advantages over other filters in removing mixed noise.
Two conclusions can be drawn from the above experiments:(1)
The improved NLM filtering algorithm proposed in this paper has excellent performance in both mixed noise removal and single noise removal. Both the visual effect and the PSNR have reached a good level with obvious advantages. However, both the AMF algorithm and the classic NLM algorithm can only filter specific noise in the image and cannot deal with the image containing complex noise.(2)
Based on the improved NLM algorithm, the precise positioning of the interactive system calibration interface is effectively improved, and the interactive point positioning is accurate, which further proves the effectiveness of the algorithm.Figure 3
Original images used in the experiment. (a) Normal grayscale. (b) Classic grayscale (Lena).
(a)(b)Table 1
Specific parameters of different filters.
Filter nameRelated parametersNLMh2 (window radius),ds (block radius)Improved NLMh2, ds (window radius), dmin, dmax, TAMFLmaxTable 2
PSNR of Lena images polluted by different salt and pepper noises.
P=0.2P=0.3P=0.4P=0.5AMF33.7530.8527.4125.51NLM25.5224.3423.8921.58Improved NLM35.6133.8531.5129.24Table 3
PSNR of ordinary human images polluted by different salt and pepper noises.
P=0.2P=0.3P=0.4P=0.5AMF34.5232.1029.1227.15NLM27.8326.2625.3523.95Improved NLM37.8535.6933.6931.58Figure 4
Lena images polluted by salt and pepper noisep=0.3 and denoising effects of different filters. (a) Original image with salt and pepper noise. (b) AMF. (c) NLM. (d) Improved NLM.
(a)(b)(c)(d)Figure 5
Ordinary human images polluted by salt and pepper noisep=0.3 and denoising effects of different filters. (a) Original image with salt and pepper noise. (b) AMF. (c) NLM. (d) Improved NLM.
(a)(b)(c)(d)Figure 6
Lena image polluted by Gaussian noiseσ=20 and denoising effects of different filters. (a) Image with Gaussian noise. (b) AMF. (c) NLM. (d) Improved NLM.
(a)(b)(c)(d)Figure 7
Ordinary human images polluted by Gaussian noiseσ=20 and denoising effects of different filters. (a) Image with Gaussian noise. (b) AMF. (c) NLM. (d) Improved NLM.
(a)(b)(c)(d)Table 4
PSNR of images with Gaussian noise after using different denoising algorithms.
LenaCharacter imageσ=15σ=20σ=25σ=30σ=15σ=20σ=25σ=30AMF28.1526.1924.5122.1328.1626.2725.5223.45NLM33.8332.5631.8630.4932.8331.9730.5330.21Improved NLM33.7532.3431.5330.1534.5633.6132.1431.89Figure 8
Lena image polluted by mixed noisep=0.2,σ=20 and denoising effects of different filters. (a) Image with Gaussian noise. (b) AMF. (c) NLM. (d) Improved NLM.
(a)(b)(c)(d)Figure 9
Ordinary human image polluted by mixed noisep=0.2,σ=20 and denoising effects of different filters. (a) Image with Gaussian noise. (b) AMF. (c) NLM. (d) Improved NLM.
(a)(b)(c)(d)Table 5
PSNR of classical grayscale images with mixed noise after using different denoising algorithms.
p=0.3p=0.4p=0.5σ=10σ=20σ=30σ=10σ=20σ=30σ=10σ=20σ=30AMF27.8525.8722.6926.7424.8521.5220.8418.8517.85NLM21.0518.7617.4819.5217.7916.3715.7014.1713.20Improved NLM29.3227.4126.6328.4126.2425.2426.7424.4722.32Table 6
PSNR of ordinary human image with mixed noise after using different denoising algorithms.
p=0.3p=0.4p=0.5σ=10σ=20σ=30σ=10σ=20σ=30σ=10σ=20σ=30AMF28.8127.1426.6925.1424.8324.2623.9123.5622.78NLM22.3822.0721.4121.2320.6420.2719.4818.2717.69Improved NLM31.9630.8329.3128.0627.8327.0426.5226.1725.93


## 4. Conclusion

To solve the problem of Gaussian and salt and pepper noise in the image processing stage of an interactive system, this paper proposes an improved NLM algorithm for filtering mixed noise. Compared with the traditional NLM algorithm, the improved algorithm can better remove the mixed noise in the image, significantly improve the PSNR of the image, and get a clearer image, which is more conducive to the interactive system to extract useful gesture or body information from the image. The next work will explore the algorithm for color image processing effect, and whether we can continue to improve the algorithm to reduce the time spent on image denoising to reduce the interaction delay.



---

*Source: 5578788-2021-07-05.xml*