<figure><img src="image_1.png" /></figure>

HAL
open science

# Application and Evaluation of Wavelet-Based Denoising Method in Hyperspectral Imagery Data

Hao Yang, Dongyan Zhang, Wenjiang Huang, Zhongling Gao, Xiaodong Yang, Cunjun Li, Jihua Wang

► To cite this version:

Hao Yang, Dongyan Zhang, Wenjiang Huang, Zhongling Gao, Xiaodong Yang, et al. Application and Evaluation of Wavelet-Based Denoising Method in Hyperspectral Imagery Data. 5th Computer and Computing Technologies in Agriculture (CCTA), Oct 2011, Beijing, China. pp.461-469, 10.1007/978-3-642-27278-3_47. hal-01361015

HAL Id: hal-01361015

https://inria.hal.science/hal-01361015

Submitted on 6 Sep 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Application and Evaluation of Wavelet-based Denoising
Method in Hyperspectral Imagery Data

Hao Yang¹, Dongyan Zhang¹,², Wenjiang Huang¹, Zhongling Gao¹, Xiaodong Yang¹,
Cunjun Li¹, Jihua Wang¹*

¹Beijing Research Center for Information Technology in Agriculture, 100097 Beijing, China;

²Institute of Agricultural Remote Sensing & Information Technology Application, Zhejiang
University, 310029 Hangzhou, China

**Abstract:** The imaging hyper-spectrometer is highly susceptible to the presence of noise and its noise removal is regularly necessary before any derivative analysis. A wavelet-based(WT) method is developed to remove noise of hyperspectral imagery data, and commonly used denoising methods such as Savitzky-Golay method(SG), moving average method(MA), and median filter method(MF) are compared with it. Smoothing index(SI) and comprehensive evaluation indicator(η) are designed to evaluate the performance of the four denoising methods quantitatively. The study is based on hyperspectral data of wheat leaves, collected by Pushbroom Imaging Spectrometer (PIS) and ASD Fieldspec-FR2500 (ASD) in the key growth periods. According to SI and η, the denoising performance of the four methods shows that WT>SG=MA>MF and WT>MA>MF>SG, respectively. The comparison results reveal that WT works much better than the others with the SI value 0.28 and η value 5.74E-05. So the wavelet-based method proposed in this paper is an optimal choice to filter the noise, in terms of balancing the contradiction between the smoothing and feature reservation ability.

**Keywords:** imaging hyper-spectrometer, noise, filtering, wavelet analysis, quantitative evaluation.

# 1 Introduction

Hyperspectral imaging system is characterized by integrating both spectroscopy and imaging techniques in one system to provide detailed information of target which otherwise cannot be achieved with either conventional imaging or spectroscopy alone[1]. The simultaneous spatial and spectral information provided by this system along with its non-destructive and chemical-free nature nominated this technology to be a deliberated tool for monitoring crop diseases[2], vegetation nutrition[3], food safety[4] etc.

But the obtained data is highly susceptible to the presence of noise. The sensors because of their narrow bandwidth are not able to capture a lot of energy and cause self-generated noise inside the sensor. Variations and atmospheric absorption of sun illumination also reduce the amount of recorded spectral signal[5]. Therefore, noise removal is regularly necessary before any derivative investigation.

Many scholars have developed different methods for how to reduce noise of spectral signal[6-9] and 2-dimension image[10-13]. At the beginning, noise reduction applied spatial average method directly, such as moving-average or median filter method based on multi-points. But it was found that in the shortwave infrared spectrum, the mean and median filter cannot remove noise satisfactorily and a slight jaggedness remains[14]. With the development of fast Fourier transform and discrete Fourier transform, noise reduction method in frequency domain[15] become acceptable. Moreover, a number of methods for hyperspectral data dimensionality reduction have been developed, such as principal component analysis (PCA)[11,16,17], linear discriminant analysis (LDA)[18] etc. Especially in

*Corresponding author. E-Mail:wangjh@nercita.org.cn
recent years, booming wavelet technology aims to provide a new way of thinking, there are
many wavelet-based and improved filter has been widely applied[7,10,19,20].

However, these denoising methods aforementioned were designed mainly for field spectrometer(ASD, e.g.), or spspaceborne and airborne hyperspectral sensors(CHRIS/PROBA, OMIS, e.g.) where noise was caused by atmosphere and water vapour etc. Rarely research about denoising of imaging hyper-spectrometer data was reported. Recently, a new field-based Pushbroom Imaging Spectrometer(PIS) is developed independently by own. So more original data can be acquired, nevertheless coupled with more noise signal, which due to physical instruments, sun's illumination, field environment condition and others. An ideal method which is simple and effective to reduce noise of the collected data is an actual need. Furthermore, as to the evaluation of denoising effect, most research is based on visual judgment or simple signal noise ratio(SNR), which always can not satisfy requirements for their complex nature.

This paper attempts to find an optimal denoising method aiming at homemade
Imaging Spectrometer (PIS). The rest of this paper is structured as follows. Section 2
describes the hyperspectral imagery data collection and introduces four denoising methods
along with the quantitative evaluation. Section 3 presents the denoising result and
evaluating result. Conclusions are drawn in Section 4.

## 2 Materials and Methods

2.1 Hyperspectral data collection

The study site located in the National Experimental Station for Precision
Agriculture, the Changping district of Beijing, China (40°11′N, 116°27′E). The cultivar
of winter wheat, Jing 411 was chosen as research material and sown in October, 2009.
When the winter wheat grown at jointing stage, 30 leaves samples were picked and
corresponding hyperspectra was collected by PIS (Pushbroom Imaging Spectrometer)
and ASD (FieldSpec®3portable spectrometer) simultaneously. The detailed process
was as follows: Put leaves flatly on the black cloth and corresponding spectrum was
measured with 120cm distance from the lens of PIS, the experimental time was from
10:30am to 15:30pm in a sunny and windless day.

PIS was developed by National Engineering Research Center for Information
Technology in Agriculture and University of Science and Technology of China. Before the
instrument was applied, the laboratory calibration had been carried out to determine
location of the wavelengths, spectral response function, radiation accuracy and spatial
displacement etc. The specific technical parameters of PIS and ASD were presented in
Table 1.

<table><caption>**Table 1.** Specific technical parameters of PIS and ASD</caption>
   <thead>
    <tr>
     <td>
      Parameters
     </td>
     <td>
      PIS
     </td>
     <td>
      ASD
     </td>
    </tr>
   </thead>
   <tbody>
    <tr>
     <td>
      Spectral Range
     </td>
     <td>
      400-1000nm
     </td>
     <td>
      350-2500nm
     </td>
    </tr>
    <tr>
     <td>
      Spectrum Resolution
     </td>
     <td>
      To 2nm
     </td>
     <td>
      3nm@700nm; 10nm@1400nm&amp;2100nm
     </td>
    </tr>
    <tr>
     <td>
      Spectrum Interval
     </td>
     <td>
      0.7nm
     </td>
     <td>
      1.4nm,350-1000nm; 2nm, 1000-2500nm
     </td>
    </tr>
    <tr>
     <td>
      Spatial Resolution
     </td>
     <td>
      To 5mm
     </td>
     <td>
      /
     </td>
    </tr>
    <tr>
     <td>
      Field of View
     </td>
     <td>
      16°
     </td>
     <td>
      25°
     </td>
    </tr>
   </tbody>
  </table>
## 2.2 Data Pre-processing

Hyperspectral imagery data can be viewed as a 3-dimensions imagery sequence, which is constituted by 2-dimensions image and 1-dimension spectrum. The 2-dimensions image describes target's spatial information feature while 1-dimension spectrum information reveals spectral profile feature of every pixel in image. For every pixel in image acquired by PIS, a sequence of digital number(DN) was obtained with spectrum interval of about 0.7nm.

For the purpose of follow-up quantitative research, DN value of wheat leaves must be transformed to relative reflectance which involved with instrument's calibration. In calibration, linear empirical method was adopted.

$$DN = a \rho + b \qquad (1)$$

In Eq.(1), the coefficient a and b was determined by experiment data and least square method. And then the reflectance ρ was calculated.

ASD data is also be transformed to reflectance through standard reference white panel.

$$\rho_{targ\_et} = (DN_{targ\_et} / DN_{panel}) \times \rho_{panel} \times 100\% \qquad (2)$$

In Eq.(2), ρ$_{targ\_et}$ and ρ$_{panel}$ mean reflectance of target and standard reference white panel, DN$_{targ\_et}$ and DN$_{panel}$ indicate DN value of target and standard reference white panel, respectively.

## 2.3 Denoising Methods

In this paper, aiming to the quantitative analysis of crop physiological and biochemical parameters, a wavelet transform method was explored to remove noise of hyperspectral imagery data, and common denoising methods such as Savitzky-Golay method(SG), moving average method(MA) and median filter method(MF) were also chosen and compared with it. Some indicators were designed to evaluate abilities of noise removal.

### 2.3.1 Wavelet Transform

Denoising techniques based on the wavelet transform, also known as wavelet shrinkage has been proposed to eliminate noise in various types of signals, by applying the transform and zeroing the coefficients below a certain threshold value, on the basis that noise coefficients would have lower amplitude than the coefficients corresponding to the feature being studied. Additionally, the coefficients remaining after the threshold operation may be reduced by the threshold value as well; this is known as soft threshold. It has been shown by Donoho and Johnstone (1994) that by using the appropriate threshold value, soft threshold results in the optimum denoising with the least distortion in the signal.

In this paper, threshold-based wavelet denoising algorithm includes mainly three steps.

1) The discrete wavelet transform and selection of wavelet filter and number of decomposition layers:

$$Y(d) = Y(f) + Y(z) \qquad (3)$$

In Eq.(3), Y(d), Y(f), Y(z) is high-frequency wavelet transform coefficient of noisy data, truth data and noise data.

2) Choose a threshold for every level high-frequency coefficient and wavelet transform coefficient Ŷ(f) was estimated by the threshold. Soft threshold and hard threshold can be used such as Eq.(4) and Eq.(5). T is threshold in them.
$$ \text{For soft threshold: } \hat{Y}(f) = \begin{cases} s \operatorname{sgn}(Y(d(\mathcal{Y})d(-\mathcal{T})) |Y|) d |\ge T \\ 0 & |Y(d)|<T \end{cases} \quad (4) $$

$$ \text{For hard threshold: } \hat{Y}(f) = \begin{cases} Y(d) & |Y(d)| \ge T \\ 0 & |Y(d)| < T \end{cases} \quad (5) $$

3) Wavelet reconstruction $\hat{Y}(f)$ to get estimation of $\hat{f}(t)$:

$$ \hat{f}(t) = \omega^{-1} \hat{Y}(f) \quad (6) $$

In Eq.(6), ω-1 is the operator of wavelet inverse transformation.

The wavelet transform filter method lies in the choice of the threshold scheme, commonly used scheme is Rigrsure, Sqtwolog, Heursure and Minimaxi. In them, Rigrsure selects an adaptive threshold based on Stein unbiased likelihood principle, Sqtwolog adopts fixed threshold $\sqrt{2 \times \log(length(x))}$, Heursure is a mixture of the first two, Minimaxi selects the mean square extreme error under non-ideal circumstance as the threshold.

The denoising effect is influenced by threshold adjustment ways. If the adjustments, there are two ways: one is that threshold is adjusted according to the first level noise(global threshold), while the other one is according to each level noise(hierarchy threshold).

### 2.3.2 Savitzky-Golay filter, Moving average, Median filter

The method of Savitzky-Golay filtering for hyperspectral analysis is discussed in detail by King et al. (1999). Savitzky-Golay filtering has its roots in least squares polynomial smoothing which is to find a smoothed value for each point in the spectrum of a subset of data within a window. The window contains the point to be smoothed in the center position of the window as well as several of its neighbors to either side in the spectrum. All the data within the window are used to perform the least squares fit, but only the central point is smoothed for each window position. The other points are smoothed by moving the window across the spectrum point by point, performing a least squares approximation to the windowed data at each location.

Moving average takes the mean spectral value of all points within a specified window as the new value of the middle point of the window(Tsai and Philpot, 1998). The method is solely based on linear calculations and has one key parameter, the filter size. The median filter is similar to moving average but it is median value not average value.

## 2.4 Evaluating Methods

Smoothing methods may cause some loss of information or change the original spectral so they will produce incorrect results in subsequent analyses such as red edge discrimination. Thus, smoothing techniques besides being powerful in the smoothing ability, should be used in the way that preserves the absorption and local spectral features like minima, maxima and inflection points. Fine filtering method evaluation criteria should be that the denoised spectral curve remains curve characteristics (curve shape and location of characteristics), and is as smooth as possible.

A smoothing index(SI) was designed to measure the smoothness of denoised spectral curve. In Eq.(7), n is number of wavebands, $\hat{\rho}$, $\rho$ is the denoised spectrum and original spectrum respectively; the SI is smaller, it means the denoised spectrum is smoother.

$$ \text{Smoothing index: } SI = \frac{\sum_{i=1}^{n} |\hat{\rho}_{i+1} - \hat{\rho}_{i}|}{\sum_{i=1}^{n} |\rho_{i+1} - \rho_{i}|} \quad (7) $$
Signal-to- Noise Ratio(SNR), Mean Square Error(MSE), Spectral Angle(SA), were also be used to evaluate smoothing effect. As the denoising effect is proportional to SNR while MSE is inversely proportional to SA, we designed a comprehensive evaluation indicator to ensue the consistency of the aforementioned three indicators. They are defined as follows:

$$ \text{Mean Square Error: } MSE = n^{-1} \sum_{i=1}^{n} (\hat{\rho}_{i} - \rho_{i})^{2} \qquad (8) $$

$$ \text{Signal-to-Noise Ratio: } SNR = 10 \cdot \log \frac{\sum_{i=1}^{n} \rho_{i}^{2}}{\sum_{i=1}^{n} (\hat{\rho}_{i} - \rho_{i})^{2}} \qquad (9) $$

$$ \text{Spectral Angle: } SA = \arccos \frac{\sum_{i=1}^{n} \rho_{i} \hat{\rho}}{\sqrt{\sum_{i=1}^{n} \rho_{i}^{2} \cdot \sum_{i=1}^{n} \hat{\rho}_{i}^{2}}} \qquad (10) $$

$$ \text{Comprehensive evaluation indicator: } \eta = \frac{MSE \cdot SNR}{SNR} \qquad (11) $$

In above equations, $\hat{\rho}$, $\rho$ is the denoised spectrum and original spectrum, n is number of wavebands respectively.

## 3 Results and analysis

The result of applying wavelet transform denoising together with other methods was demonstrated in Figure 1. We chosen an area in the middle of leave and got its DNs average in the area in every band, so an average spectrum was acquired. In fact, this process meant the correlation between pixels in spatial domain is utilized. The average spectra were showed in blue curves in Figure 1. Red curves displayed the results after noise removal. All algorithm was realized in MATLAB platform. In wavelet transform filter, sqtwolog, soft threshold, hierarchy threshold scheme were selected, the number of decomposition layers was set to 5, wavelet radix chosen “symlets20”. For SG filter, the number of envelope iterations was 2 and window size was 5. And in MA and MF filter, the filter step was 5.

For four methods, the effects of noise removal can be seen in the Fig.1. The obvious noises were removed and the whole shapes of spectral curves were preserved. When compared to the other, WT was smoother than others, especially in wavelength greater than 480nm. Contrasting the curves in 750-950nm, some small disturbances remained in Fig.1(b,c,d) while Fig.1(a) was smooth.

Evaluating methods in section 2.4 were applied to further assess the effect of denoising quantitatively. The result was showed in Table 2. In Table 2, two critical indicators should be concerned: SI and $\eta$. SI represents smoothness, the less means the smoother. WT had the minimum value of 0.28, MF had the maximum of 0.36. The denoising effect was WT>SG=MA>MF. $\eta$ measured denoising ability from another view. Similarly, WT had the minimum value of 5.74E-05, it meant WT had the best performance. The effect was WT>MA>MF>SG. As a result, both of evaluating indicators showed WT had the best denoising performance.

Above result was based on region of pixels. For single pixel, the result of WT filter was showed in Fig.2. The red curve represented denoising spectrum. It can be found WT was still very efficient though much noise existed in this case. So the WT filter was stable
in denoising, especially in >480nm wavelength range.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** The results of four denoising methods</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Figure 2.** Denoising result of single pixel by WT **Figure 3.** Denoising Spectra of ASD and PIS by WT</figcaption></figure>

<table><caption>**Table 2.** The statistics of evaluation results for four methods</caption><thead><tr><th>Indicator</th><th>SI</th><th>SNR</th><th>MSE</th><th>SA</th><th>η</th></tr></thead><tbody><tr><td>WT</td><td>0.28</td><td>36.56</td><td>0.14</td><td>0.015</td><td>5.74E-05</td></tr><tr><td>SG</td><td>0.35</td><td>35.12</td><td>0.20</td><td>0.018</td><td>1.03E-04</td></tr><tr><td>MA</td><td>0.35</td><td>36.15</td><td>0.16</td><td>0.016</td><td>7.08E-05</td></tr><tr><td>MF</td><td>0.36</td><td>35.65</td><td>0.18</td><td>0.016</td><td>8.08E-05</td></tr></tbody></table>

When the selected region from PIS image was same to the view field of ASD, the same target was measured. Owing to filter algorithm was built-in ASD, the obtained ASD spectrum had little noise and can be viewed as reference spectrum. Denoised PIS spectrum by WT and the corresponding ASD spectrum was showed in Fig. 3. As shown, the former
approach to the later. The trend was very consistent though absolute reflectance differences.
The absolute reflectance difference can be explained by the two instrument sensitivity itself,
illumination condition, calibration accuracy, and time etc.

# 4 Discussion and conclusions

This study presented an approach to removing noise of hyperspectral imagery data which were collected by PIS. Aiming to the quantitative inversion of crop physiological and biochemical parameters, the wavelet-based method was developed, and commonly used denoising methods such as Savitzky-Golay method(SG), moving average method(MA), and median filter method(MF) were compared with it. And a smoothness index and a comprehensive evaluation indicator were designed to appraise denoising effects. The denoised effects of four methods were assessed by them. For index SI, the denoising effect was WT>SG=MA>MF; for η, the result was WT>MA>MF>SG. So WT had the best noise removal performance. The denoising result by WT was also compared with ASD data, the effect was close to each other. As a result, WT filter was a very efficient means to removing noise for hyperspectral imagery data.

Though WT method showed powerful denoised ability, there was space for further
research. Firstly, this paper focused mainly on spectrum dimension, and put little
consideration on image dimension noise reduction. Secondly, for evaluation of denoised
effect, the characteristic preservation ability can be evaluated through detailed key feature
such as location of small peak and valley which depends on specific application. Denoising
data was applied to biochemical parameters inversion, and then best denoising method was
evaluated. In addition, the factors, such as illumination, temperature, wind, and human are
all affect the experimental results, so how to ensure optimal spectrum information that
reflects truth need further to study.

Acknowledgements

This work was subsidized by the Beijing Natural Science Foundation (Project Number:
4092016), China Special Funds for Major State Basic Research (Project Number:
2007CB714406, 2011CB311806), and China Postdoctoral Science Foundation funded
project (Project Number: 20110490317). The authors are grateful to Mr. Weiguo Li, and
Mrs. Hong Chang for data collection.

References

[1]. Wang J H, Zhao C J, Huang W J. Basis and Application of Quantitative Remote Sensing in Agriculture[M]. Beijing: Science Press, 2008:141-184. (in Chinese)

[2]. C.B. Singh, D. S. Jayas, J. Paliwal, N.D.G. White. Detection of insect-damaged wheat kernels using near-infrared hyperspectral Imaging. Journal of Stored Products Research, 2009, 45, 151-158.

[3]. N. Nguyen Do Trong, M. Tsuta, B.M. Nicola, J. De Baerdemaeker, W. Saeys. Prediction of optimal cooking time for boiled potatoes by hyperspectral imaging. Journal of Food Engineering, 2011, 105, 617-624.

[4]. L. Liu, M.O. Ngadi S.O. Prasher, C. Gariépy. Categorization of pork quality using Gabor filter-based hyperspectral imaging technology. Journal of Food Engineering, 2010, 99, 284-293.

[5]. Mohd Rozni M. Yusof. Trends and Issues in Noise Reduction for Hyperspectral Vegetation Reflectance Spectra. European Journal of Scientific Research, 2009, 29(3): 404-410.
[6]. Wang Ying, Mo Jinyuan. A New De-Noising Technique for Spectra Based on Mexican Hat Wavelet. Spectroscopy and Spectral Analysis, 2005, 25(1):124-127.

[7]. Zhou Dan, Wang Qinjun, Tian Qingjiu, Lin Qizhong, Fu Wenxue. Wavelet Analysis and Its Application in Denoising the Spectrum of Hyperspectral Image. Spectroscopy and Spectral Analysis, 2009, 29(7), 1941-1945.

[8]. Baoxin Hu, Qingmou Li, A. Smith. Noise reduction of hyperspectral data using singular spectral analysis. International Journal of Remote Sensing, 2009, 30(9), 1954-1957.

[9]. Longcun Jin, Wanggen Wan, Yongliang Wu, Bin Cui, Xiaoqing Yu. A General Framework for High-Dimensional Data Reduction Using Unsupervised Bayesian Model. Communications in Computer and Information Science, 2010, 98(2), 96-101.

[10]. Sun Lei, Gu De-feng, Luo Jian-shu. Hyperspectral Imagery Denoising Method Based on Wavelets. Spectroscopy and Spectral Analysis, 2009, 29(7), 1954-1957.

[11]. Guangyi Chen, Shenen Qian. Simultaneous dimensionality reduction and denoising of hyperspectral imagery using bivariate wavelet shrinking and principal component analysis. Can.JRemote Sensing, 2008, 34(5): 447-454.

[12]. L. Gómez-Chova, L. Alonso, L. Guanter, G. Camps-Valls, J. Calpe, J. Moreno. Correction of systematic spatial noise in push-broom hyperspectral sensors: application to CHRIS/PROBA images. APPLIED OPTICS, 2008, 47(28), 46-60.

[13]. Nguyen Quang Minh. Image smoothing of multispectral imagery based on the HNN and geo-statistics. Journal of Remote Sensing, 2011, 15(3): 640-644.

[14]. Schmidt, K. S. and Skidmore, A. K. Smoothing vegetation spectra with wavelets. International Journal of Remote Sensing, 2004, 25(6): 1167-1184.

[15]. Atkinson, I., Kamalabadi F., Jones D. L. Wavelet-based hyperspectral image estimation. Proceedings of International Geoscience and Remote Sensing Syposium. 2003, 2, 743-745.

[16]. Jolliffe, T. 2002. Principal component analysis. Springer, New York.

[17]. Chang Weiwei, Guo Lei, Liu Kun, Fu Chaoyang. A noise removal method for hyperspectral data based on Contourlet transformation and PCA analysis. Journal of Electronics & Information Technology, 2009, 31(12), 2892-2896.

[18]. Fukunaga, K. 1990. Introduction to statistical pattern recognition. Academic Press, San Diego, Calif.

[19]. Othman, H., Qian, S.-E. Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage. IEEE Transactions on Geosciences and Remote Sensing, 2006, 44, 397-408

[20]. Huang Mingxiang, Wang Ke, Shi Zhou, Gong Jianhua, Li Hongyi, Chen Jieliang. Quantitative Evaluation of Soil Hyperspectra Denoising with Different Filters. Spectroscopy and Spectral Analysis, 2009, 29(3), 722-725.