Preprocessing of hyperspectral imagery withconsideration of smile and keystone properties
Naoto Yokoya*a, Norihide Miyamurab and Akira Iwasakib
Dept. of Aeronautics and Astronautics, The University of Tokyo, Japanb
Research Center for Advanced Science and Technology, The University of Tokyo, Japana

ABSTRACT
Satellite hyperspectral imaging sensors suffer from ‘’smile’’ and ‘’keystone’’ properties, which appear as distortions ofspectrum images. The smile property is a center wavelength shift and the keystone property is a band-to-bandmisregistration. These distortions degrade the spectrum information and reduce classification accuracies. Furthermore,these properties may change after the launch. Therefore, in the preprocessing of satellite hyperspectral images, theonboard correction of the smile and keystone properties is an important issue as well as the radiometric and geometriccorrection. The main objective of this work is to propose the prototype of the preprocessing of hyperspectral image withconsideration of smile and keystone properties. Image registration based on phase correlation is used for detecting theoptical properties. Cubic spline interpolation is adopted to modify the spectrum because of its good trade-off between thesmoothness and shape preservation. Smile and keystone detection simulation using the EO-1 Hyperion imagery taken atvarious times in the past nine years proved that the optical properties have been changing due to the onboard seculardistortion. Therefore, onboard optical properties should be updated periodically and built into the radiometric andgeometric corrections for future satellite hyperspectral sensors. The proposed method may be the prototype of the
preprocessing of future satellite hyperspectral sensors.
Keywords: hyperspectral imagery, smile, keystone

1. INTRODUCTION
Hyperspectral imagers are pushbroom sensors that continuously collect spectrograms that form a three-dimensional (3D)
data cube, composed of cross-track (x), along-track (y), and spectrum (λ) dimensions, as shown in Figure 1.
Hyperspectral imagery contains hundreds to thousands of spectral bands. At each pixel, a continuous spectral profile canbe obtained, which enables discrimination among land-cover classes that are spectrally similar [1]. Many spaceborne
hyperspectral sensors are developed as the next generation advanced earth observing satellite, such as EnMap (German),
PRISMA (Italy), HyspIRI (USA), and HISUI (Japan) [2,3,4,5].

Figure 1. Hyperspectral imaging*yokoya@sal.rcast.u-tokyo.ac.jp; phone +81-3-5452-5319

Hyperspectral imageries suffer from sensor optical properties, which distort spectrograms due to optical aberrations andmisalignments. These spectrographic distortions cause spectral and spatial misregistrations, commonly called as “smile”
and “keystone” properties (Figure 2). Smile property is a shift in wavelength in the spectral domain and keystone
property is a band-to-band misregistration. They distort spectral profiles and thus degrade classification results. For
example, maximum noise fraction (MNF) transform [6], which produces new components ordered by the signal to noise
ratio (SNR), is commonly used for spectral dimensionality reduction of hyperspectral imagery before classification,
however, smile effect appears in the first MNF component as a brightness gradient [7]. This artifact is dominant inclassification and leads to misclassification [8]. Past studies demanded very high accuracy of optical properties [9,10].

Figure 2. Smile and keystone properties

The Hyperion imaging spectrometer is the first spaceborne hyperspectral sensor that routinely acquires science-grade
data mounted on the Earth Observing 1 (EO-1) satellite, which was launched on November 21, 2000. This sensor
collects data with a 30 m ground sampling distance over a 7.65-km swath in 198 spectral bands with two spectrometers;
the first is the visible near-infrared radiometer (VNIR) over 400-1000 nm, and the other is the shortwave infrared
radiometer (SWIR) over 900-2500 nm. The Hyperion data is freely available at the website of U. S. Geological Survey(USGS) (http://edcsns17.cr.usgs.gov/EarthExplorer/). Two data formats, L1R and L1Gst, are available. L1R data isradiometrically corrected and L1Gst data is radiometrically corrected and resampled for geometric correction andregistration to a geometric map projection with ortho-rectification. Hereafter, we refer to L1Gst as L1G. The Hyperion
data contains various noises such as stripe and shot noise and also suffers from smile and keystone properties. Due tothese artifacts, classification performance using the Hyperion data was outperformed by those using multispectral data insome cases [11,12]. The prelaunch smile property is attached to the data but the prelaunch keystone property is not.
Smile and keystone properties are not corrected in both types of data, L1R and L1G. Moreover, many studies showedthat smile property may have changed after the launch. Therefore, building an accurate onboard calibration of opticalproperties into the L1R and L1G data processing is an important and challenging issue.
The postlaunch smile property was estimated by correlating the sensor-measured spectrum with a spectrum model using
moderate resolution atmospheric transmission (MODTRAN) [13,14]. The postlaunch keystone property was detected bya scene-based method based on edge detection using a sharpening filter [15,16]. However, in these past studies, theestimation accuracy and robustness for spatially and temporally different scenes have not yet been evaluated. Recently,
we have proposed a methodology for the detection and correction of smile and keystone properties only from observed
data [17]. This method is proved to be accurate and robust for various scenes. In this paper, we propose the prototype forthe preprocessing of hyperspectral imagery, which produces L1R and L1G data, building in onboard calibration of smileand keystone properties. The Hyperion VNIR data is used as an example. The effect of the onboard calibration of opticalproperties on L1G data is also demonstrated.

2. OVERVIEW FOR L1R AND L1G PROCESSING
Figure 3 shows the flow chart of L1R and L1G data processing which detect and correct the artifacts in hyperspectral
imagery. Stripe noise and optical properties are all related to a two-dimensional (2D) detector array and pushbroom
imaging system. After the geometric correction, the relationship between the 2D spectroscopic image and the 3D
hyperspectral image cube has disappeared. Thus, the removal of these artifacts must be processed before the geometriccorrection. Since resampling used for data correction degrades spectral signatures, to minimize resampling frequency,
smile and keystone correction should be done with radiometric and geometric corrections, respectively.

Figure 3. L1R and L1G data processing with consideration of smile and keystone properties.

In the following sections, improvement of radiometric coefficient with stripe noise reduction and accurate onboard
calibration of smile and keystone properties are described in details.

3. RADIOMETRIC IMPROVEMENT WITH STRIPE NOISE REDUCTION
In the Hyperion imagery, there exist a lot of stripe noises which reduce data quality and interpretability. Almost all ofthese stripe artifacts are of one pixel width. Several destriping methods have been proposed [18,19], however, thesemethods are complicated and difficult to replicate accurately. We use the simplest methods for both detection andcorrection of stripe noise. Stripe noise is a line whose brightness is higher or lower than neighboring lines. Therefore, we
detect it using a limit check. Consider a 3D hyperspectral data cube, in which each component located in band k at crosstrack i and along-track j position is represented by xi , j ,k ( 1  i  W , 1  j  H , 1  k  B ). Here, W and H are the widthand height and B is the total number of bands, respectively. In a certain cross-track position i in band k, if the total
number of pixels, whose brightness is higher or lower than both of cross-track neighboring two pixels, is over a threshold,the line at this cross-track position is detected as a stripe. The total number N i , j , k is given by

N i , k   j ci , j ,k ,
where

1
ci , j ,k  0( xi , j ,k  xi 1, j ,k , xi 1, j ,kotherwiseorxi , j ,k  xi 1, j ,k , xi 1, j ,k ) .

Three sigma limits are used for the threshold. After stripe detection, a moment matching method is applied for stripe
noise correction [20]. Moment matching calculates the improved radiometric gain and offset to adjust the average andstandard deviation of line radiance to that of the reference line, which is set as the average of neighboring two lines. Each
DN is adjusted as follows:

DN adjusted  ( DN initial   i ,k )

 reference  reference .
 i ,kHere,  i,k and  i,k denote the average and standard deviation of line radiance located in band k at cross-track i, and reference and  reference denote those of the reference line. This detection and correction process is repeated until no more
stripes are detected. Figure 4 shows the original image, detected stripe, and corrected image for the Hyperion band 57

Figure 4. Original image (left), detected stripe (middle), and corrected image (right).

image taken over a desert in Chile. After stripe noise reduction, data quality has improved. Since stripe noise deteriorates
smile and keystone detection results, radiometric improvement with stripe noise reduction should be processed before
calibration of optical properties.

4. ONBOARD CALIBRATION OF SMILE AND KEYSTONE PROPERTIES
4.1 Detection and correction methods
Image registration was applied for the accurate detection of onboard optical properties only from observed data [17]. By
estimating spectroscopic image distortions in spectral and spatial directions, smile and keystone properties can bedetected. In this work, we use image registration based on phase correlation (PC). Consider two N1  N 2 images
f (n1 , n2 ) and f (n1   1 , n2   2 ) that differ only by displacement ( 1 ,  2 ) . To reduce the image boundary effect in thefrequency domain, a 2D Hanning window is applied to the input image. A low-pass filter is also applied in the frequency
domain to eliminate the high-frequency components of image. The simplest low-pass filter is given
by H (1 ,  2 )   1  1  U 1 , 1   2  U 2 otherwise 0  . In this case, the phase correlation r (n1 , n2 ) is
simplified bysinr (n1 , n2 )   V1V (n1   1 ) sin 2  (n2   2 )N1
N2, (n1   1 ) (n2   2 )where   1 , V1  2U 1  1 and V2  2U 2  1 . The value of displacement ( 1 ,  2 ) and the value  can be obtained
independently, assuming that one misregistration is fixed. Thus, for mathematical simplicity, an 1D version of thecorrelation function is considered in the following description. The 1D correlation function is given byV
sin  (n   )N.
r ( n)   
 (n   )The displacement can be obtained in the subpixel level by estimating the peak location of the correlation function. Usingsinc function fitting, the subpixel value of the displacement is directly obtained byr (n0  1)  r (n0  1)
,

V


r (n0  1)  2 cos  r (n0 )  r (n0  1)
 N
where n0 is the integer with the maximum value of correlation function [21].

For the smile and keystone correction, we use cubic spline interpolation because of its good trade-off between
smoothness and shape preservation [22]. Sinc function interpolation coded in MATLAB with a not-a-knot end condition
is applied to hyperspectral data cube in the spectral and cross-track directions to modify smile and keystone properties,
respectively.
4.2 Smile property
Smile effect was estimated by applying image registration based on phase correlation to a spectroscopic image in the
cross-track direction. The size of correlation window ( Lx , L ) is set as (1,11), where Lx and L are the lengths in crosstrack and spectral direction, respectively. Since bands 1-7 and 58-70 of the Hyperion imagery have no data andcorrelation window is wide in the spectral direction, we estimate smile effect in bands 13-52. Nine scenes taken over
Sahara desert in 2002 to 2010 are used as examples because spectral signature should be homogeneous in cross-track
direction for smile detection. We estimate smile effect for each along-track position and return the average result as the
final smile effect. Figure 5 shows the estimated smile effect of bands 13-52 for nine desert scenes. The bottom number
denotes the data acquisition date; the first number is the year and the second is the number of days elapsed from January
1st. Smile property can be detected when a spectral signature in the correlation window contains a clear atmosphericabsorption line [17]. Therefore, smile property in band 41, which is an oxygen absorption line, can be detected. Smile
property can be approximated as a quadratic function. Figure 6 shows secular change appearing in cross-track position inband 41 smile axis. Smile property has been changing after the launch with a certain tendency. It indicates that periodical
update of smile property is necessary for satellite hyperspectral sensor. According to MNF transform analysis, the smile
effect in band 41 clearly appears in the difference image between bands 40 and 42. Figure 7 shows comparison of thisdifference image between before and after smile effect correction. Smile brightness gradient in the spectral direction
disappears after smile correction. Figure 8 shows the difference image between bands 40 and 42 in L1G data format
taken over San Francisco. Brightness gradient also disappears in L1G, which shows that smile correction before
geometric correction improved the quality of L1G data. Thus, the L1G data product can be distributed to users after
smile correction. There is no or little smile effect in bands 13-34. It may be because radiometric correction in bands 1334 was not carried out accurately. It suggests that radiometric coefficient and smile property should be iteratively
estimated for a number of times.

Figure 5. Smile effect estimated for various nine imagery taken over Sahara desert in 2002-2010.

Figure 6. Secular change of band 41 smile property appearing in cross-track position of smile axis.

Figure 7. Radiance difference images between bands 40 and 42 before (upper images) and after (lower images) smile correction forvarious nine imagery taken over Sahara desert in 2002-2010.

Figure 8. Radiance difference between bands 40 and 42 before (middle) and after (right) smile correction in L1G data.

4.3 Keystone property
Keystone property was detected at each cross-track position by applying image registration to a small hyperspectral cubein the spectral direction and estimating band-to-band misregistrations. The size of the correlation window ( Lx , L y ) is set
to (31,31), where Lx and L y are the lengths in cross-track and along-track direction, respectively. Phase correlation
method assumes that input and reference images are almost the same. Desert scenes with clear texture were used as
examples because there were no contrast conversions such as vegetation, and texture of each band was almost the same.
The average result of detected keystone properties at all along-track positions was defined as the final detected keystone
property. Figure 9 shows the comparison of images before and after keystone property detection for nine desert scenes
also used for smile property, especially using imagery taken over ripple marks. The Maximum band-to-bandmisregistration that was about 10-20% of spatial resolution before keystone correction reduced to within 5%. The
Hyperion keystone property can be approximated as a linear function. Figure 10 shows the secular change of keystoneslope. There is also a certain tendency consistent with that of smile property. It strongly suggests that optical propertiesof the Hyperion may have varied across the days after the launch. Therefore, an onboard periodical update of opticalproperties built into L1R and L1G data processing is necessary for future satellite hyperspectral sensors.

Figure 9. Keystone properties before (upper images) and after (lower images) keystone correction estimated for various nine imagery
taken over Sahara desert in 2002-2010.

Figure 10. Secular change of keystone slope.

5. CONCLUSION
In this work, we proposed the L1R and L1G data processing of hyperspectral imagery with consideration of smile andkeystone properties. Image registration based on phase correlation is used for detection of smile and keystone properties
and cubic spline interpolation is applied for correction of spectral signatures. Smile correction should be built into
radiometric correction. Furthermore, keystone correction should be processed together with geometric correction. Smileand keystone detection simulation using the EO-1 Hyperion imagery taken at various times in the past nine years
demonstrated that the optical properties have been changing due to the onboard secular distortion. Therefore, onboard
optical properties should be updated periodically and built into L1R and L1G data processing for future satellite
hyperspectral sensors.

REFERENCES[1] Kruse, F. A., Boardman, J. W., and Huntington, J. F., "Comparison of airborne hyperspectral data and EO1 Hyperion for mineral mapping," IEEE Trans. Geosci. Remote Sens. 41(6), 1388-1400 (2003).
[2] Sang, B., Schubert, J., Kaiser, S., Mogulsky, V., Neumann, C., Förster, K., Hofer, S., Stuffler, T.,
Kaufmann, H., Müller, A., Eversberg, T., Chlebek C., "The EnMAP hyperspectral imaging spectrometer:
instrument concept, calibration and technologies, " Proc. SPIE 7086, 708605 (2008).
[3] Galeazzi, C., Sacchetti, A., Cisbani, A., Babini, G., "The PRISMA program," Proc. IEEE International
Geoscience and Remote Sensing Symposium 4, 105-108 (2008).
[4] Chien, S., Silverman, D., Davies, A. G., Mandl, D., "Onboard science processing concepts for the HyspIRI
mission," IEEE Intelligent Systems 24(6), 12-19 (2009).
[5] Ohgi, N. and Iwasaki A., "Japanese hyper-multi spectral mission," Proc. IEEE International Geoscienceand Remote Sensing Symposium (2010).
[6] Green, A. A., Berman, M., Switzer, P., and Craig, M. D., "A transformation for ordering multispectral datain terms of image quality with implications for noise removal," IEEE Trans. Geosci. Remote Sens. 26(1),65-74 (1988).
[7] Goodenough, D. G., Dyk, A., Niemann, K. O., Pearlman, J. S., Chen, H., Han, T., Murdoch, M., and West,
C., "Processing Hyperion and ALI for forest classification," IEEE Trans. Geosci. Remote Sens. 41(6),1321-1331 (2003).

[8] Dadon, A., Ben-Dor, E., and Karnieli, A., "Use of derivative calculations and minimum noise fraction
transform for detecting and correcting the spectral curvature effect (smile) in Hyperion images," IEEE
Trans. Geosci. Remote Sens. 48(6), 2603-2612 (2010).
[9] Green, R. O., "Spectral calibration requirement for Earth-looking imaging spectrometers in the solarreflected spectrum," Appl. Opt. 37(4), 683-690 (1998).
[10] Mouroulis, P., Green, R. O., and Chrien, T. G., "Design of pushbroom imaging spectrometers for optimum
recovery of spectroscopic and spatial information," Appl. Opt. 39(13), 2210-2220 (2000).
[11] Waldhoff, G., Bubenzer, O., Bolten, A., Koppe, W., and Bareth, G., "Spectral analysis of ASTER,
Hyperion, and QuickBird data for geomorphological and geological research in Egypt (Dakhla Oasis,
Western Desert)," in Proc. XXI ISPRS Congress, Commission VIII, XXXVIII-B8, 1201-1206 (2008).
[12] Gordley, L., Marshall, B., McHugh, M., Fritts, D., and Fish, C., "Spectral analysis of ASTER and Hyperion
data for geological classification of Volcano Teide," Proc. IEEE International Geoscience and Remote
Sensing Symposium (2010).
[13] Pearlman, J. S., Barry, P. S., Segal, C. C., Shepanski, J., Beiso, D., and Carman, S. L., "Hyperion, a spacebased imaging spectrometer," IEEE Trans. Geosci. Remote Sens. 41(6), 1160-1173 (2003).
[14] Neville, R. A., Sun, L., and Staenz, K., "Detection of spectral line curvature in imaging spectrometer data,"
Proc. SPIE 5093, 144-154 (2003).
[15] Neville, R. A., Sun, L., and Staenz, K., "Detection of keystone in imaging spectrometer data," Proc. SPIE
5425, 208-217 (2004).
[16] Dell'Endice, F, Nieke, J., Schlapfer, D., and Itten, K. I., "Scene-based method for spatial misregistration
detection in hyperspectral imagery," Appl. Opt. 46(15), 2803-2816 (2007).
[17] Yokoya, N., Miyamura, N., and Iwasaki, A., "Detection and correction of spectral and spatial
misregistrations for hyperspectral data using phase correlation method," Appl. Opt. 49(24), 4568-4575(2010).
[18] Tsai, F., and Chen, W. W., "Striping noise detection and correction of remote sensing images," IEEE Trans.
Geosci. Remote Sens. 46(12), 4122-4131 (2008).
[19] Sun, L., Neville, R. A., Staenz, K., and White, H. P., "Automatic destriping of Hyperion imagery based on
spectral moment matching," Can. J. Remote Sens. 34(1), S68-S81 (2008).
[20] Gadallah, F. L., Csillag, F. and Smith, E. J. M., "Destriping multisensor imagery with moment matching,"
Int. J. Remote Sens., 21(12), 2505–2511 (2000).
[21] Nagashima, S., Aoki, T., Higuchi, T., and Kobayashi, K., "A subpixel image matching technique using
phase-only correlation," Proc. IEEE International Symposium on Intelligent Signal Processing andCommunication Systems, 701-704 (2006).
[22] Feng, Y. and Xiang, Y., "Mitigation of spectral mis-registration effects in imaging spectrometers via cubic
spline interpolation," Opt. Express 16(20), 210-221 (2002).