HYPERSPECTRAL AND MULTISPECTRAL DATA FUSION MISSIONON HYPERSPECTRAL IMAGER SUITE (HISUI)
Naoto Yokoya and Akira Iwasaki
Research Center for Advanced Science and Technology, The University of Tokyo, Japan
yokoya@sal.rcast.u-tokyo.ac.jp
ABSTRACT
Hyperspectral imager suite (HISUI) is the Japanese nextgeneration earth-observing sensor composed of hyperspectral
and multispectral imagers. Unmixing-based fusion of hyperspectral and multispectral data enables the productionof high-spatial-resolution hyperspectral data. HISUI simulated imaging system combining two imagers was developedfor verification experiments to investigate the feasibility andclarify the whole procedure of the hyperspectral and multispectral data fusion mission on HISUI. Airborne experiments are planned as simulation tests of HISUI higher-orderproducts. The experimental results of the ground based observation showed the importance of the preprocessing andcross-calibration on the final quality of fused data, which
contributes to the practical use of hyperspectral and multispectral data fusion.
Index Terms— Hyperspectral and multispectral data fusion, unmixing, hyperspectral imager suite (HISUI)
1. INTRODUCTION
Hyperspectral imagers (HSIs) collect approximately 100 to
200 or more spectral bands with 5–10 nm spectral bandwidths, whereas multispectral imagers (MSIs) obtain approximately 4 to 10 spectral bands with larger spectral bandwidths(70–400 nm). HSIs generally have larger ground sampling
distance (GSD) than MSIs due to a trade-off of sensor design between spatial and spectral resolutions. Hyperspectral
imager suite (HISUI) is the Japanese next-generation earthobserving sensor composed of hyperspectral and multispectral imagers [1]. The GSDs of hyperspectral and multispectral
imagers are 30 m and 5 m, respectively. In the visible nearinfrared spectral range, the HSI has 57 bands, whereas theMSI has 4 bands.
Hyperspectral and multispectral data fusion can produce high-spatial-resolution hyperspectral data [2, 3, 4, 5].
Unmixing-based hyperspectral and multispectral data fusion
can enhance the spatial resolution of hyperspectral data witha small spectral distortion [4, 5]. An unmixing-based hyperspectral and multispectral data fusion method, named couplednonnegative matrix factorization (CNMF), was proposed inremote sensing [4]. CNMF is composed of alternating unmixing for two images using nonnegative matrix factorization(NMF) [6]. NMF has recently been receiving attention in unmixing of remotely sensed hyperspectral data based on alinear spectral mixture model to deal with severe mixturesconsidering nonnegativity with simple implementation [7, 8].
A similar approach for resolution enhancement of hyperspectral data using an RGB image was independently proposed incomputer vision [5]. This method uses a sparsity promoted
unmixing that is suitable for computer vision applications.
Several assumptions for this data fusion are as follows:
1) observing from the same platform with the same observation conditions such as atmospheric and illumination
conditions, 2) observing the same scene with accurate image
registration, 3) relative sensor characteristics such as spectral
response functions (SRFs) and point spread functions (PSFs)
are given. Hyperspectral and multispectral data fusion algorithms in remote sensing have mainly been evaluated usingsynthetic datasets because there is no platform that is composed of hyperspectral and multispectral imagers with thetrade-off of spatial and spectral resolution. HISUI will be
the first spaceborne platform that satisfies these assumptions.
Verification experiments are indispensable to clarify the practical issues and establish the whole operation including thepreprocessing and cross-calibration. In this work, we present
a high-spatial-resolution hyperspectral imaging system combining hyperspectral and multispectral imagers and show its
ground experiment to investigate the feasibility of the hyperspectral and multispectral data fusion mission on HISUI.
The plan of airborne experiments is also described as further
simulation tests.
2. UNMIXING-BASED HYPERSPECTRAL AND
MULTISPECTRAL DATA FUSION
2.1. The CNMF algorithm
In this work, CNMF is used for HISUI hyperspectral andmultispectral data fusion mission owing to its advantage forremote sensing data. The aim of hyperspectral and multispectral data fusion is to estimate unobservable high-spatialresolution hyperspectral data (Z 2 RLh ⇥Pm ) from observable

X = ZS and Y = RZ,(1)

where S 2 RPm ⇥Ph and R 2 RLm ⇥Lh are the relative PSF
and SRF matrices, respectively. With the linear spectral mixture model, high-spatial-resolution hyperspectral data can beexpressed asZ ⇡ EA,(2)where E 2 RLh ⇥D and A 2 RD⇥Pm denote the endmember
and abundance matrices, respectively, and D is the number ofendmembers. The high-spatial-resolution hyperspectral data
can be obtained by estimating the endmember and abundance
matrices from observed two datasets. First, the algorithm
starts from NMF-based unmixing of the hyperspectral datato estimate the endmember spectra taking its spectral advantage. Next, the multispectral data is unmixed by NMF after
initializing the endmember and abundance matrices using theunmixing results of the hyperspectral data. The sequential
unmixing for hyperspectral data is processed after initializing the abundance fractions by using the unmixing results ofthe multispectral data. After that, two data are alternately unmixed until convergence and the fused data can be obtainedby multiplying the endmember matrix by the high-spatialresolution abundance matrix. NMF converges to a local minimum; therefore, the initialization is important. The relativeSRFs are used for initializing the endmember spectra in themultispectral unmixing and the relative PSF is used for initializing the abundance maps in the hyperspectral unmixing.
CNMF uses the advantages of hyperspectral and multispectral data, i.e., spectral and spatial resolutions, respectively, tofind better local minima of the other unmixing process. Moredetails about the CNMF method are given in [4].
2.2. Preprocessing and cross-calibration
When applied to real datasets, the preprocessing of observed
data and the cross-calibration of the two sensors are necessary to satisfy the assumptions [9]. Fig. 2 shows a flowchartof the whole process. First, the observed datasets are radiometrically corrected and co-registered in the preprocessing. Accurate image registration is critical to consistency ofabundance maps between hyperspectral and multispectral images. Histogram matching is useful for radiometric correction. Extreme noise contained in the data need to be corrected

Y=RZ
SRF

High-spa!al-resolu!onhyperspectral data

Mul!spectral data

PSFZ
X=ZSY
Unmixing…

Data
reconstruc!on…

low-spatial-resolution hyperspectral data (X 2 RLh ⇥Ph ) andhigh-spatial-resolution multispectral data (Y 2 RLm ⇥Pm )(Fig. 1). Lh and Lm denote the numbers of spectral channelsof hyperspectral and multispectral sensors, respectively. Phand Pm denote the numbers of pixels of hyperspectral andmultispectral images, respectively. All data are expressed inmatrix form with each column vector representing a spectrum
at each pixel. Lh > Lm and Ph < Pm are satisfied by
the trade-off between spectral and spatial resolutions. Hyperspectral and multispectral data can be seen as degradeddatasets of the ideal data in spatial and spectral domains:*Unmixing
X
Hyperspectral data

EEndmemberspectraAAbundance maps

Fig. 1. Illustration of unmixing-based hyperspectral and multispectral data fusion
Observed dataPreprocessing* Image registra!on* Radiometric calibra!on

Cross-calibra!on* Spectral response func!on* Point spread func!onData fusion
Fused data

Fig. 2. Flowchart of hyperspectral and multispectral data fusionin the preprocessing because it can be detected as one of endmembers and worsen unmixing results and a quality of fused
data. Next, the cross-calibration is performed to obtain relative sensor characteristics, such as SRF and PSF. The relativeSRFs, which are used for the initialization of the endmember
spectra of the multispectral unmixing, can be estimated byconstrained least squares methods that is solved by quadratic
programming [9, 10]. The relative PSF is obtained by comparing sharpnesses of hyperspectral and multispectral images
and used for the initialization of the abundance maps of thehyperspectral unmixing. Finally, the alternate unmixing of
hyperspectral and multispectral data is processed using therelative sensor characteristics for relating endmember spectra
and abundance fractions.
3. GROUND EXPERIMENT
We carried out the ground experiment of hyperspectral andmultispectral data fusion using combined imagers as shownin Fig. 3(a). The HSI is HyperSpec-VNIR-C of Headwall
photonics Inc., which uses a diffraction grating spectrometer
as illustrated in Fig. 3(b). This sensor can observe spectral
images in 12 bit over the range of 390–1040 nm with the

(i)

(b)

(i) Object lens(ii) Slit;ŝŝŝͿZĞŇĞĐƚŽƌ
(iv) Concave mirror;ǀͿŝīƌĂĐƟŽŶŐƌĂƟŶŐ
(vi) Concave mirror(vii) Detector

HSIMSI(c)(iii)

(vii)

(v)

(iv)

SAE
PSNR

HSI
1.483
31.62

MSI
0.937
42.81(vi)

Focal plane of MSI
Linesensors

ImagecircleZŽƚĂƟŽŶƐƚĂƚŐĞ

Table 1. SAE (in degrees) and PSNR (in decibels).

(ii)

B RG NIR x2

Fig. 3. (a) Combined optical imagers and illustrations of (b)HSI and (c) focal plane of MSI.

Blue
GreenRed
NIR

0.15

Rela!ve SRF(a)

0.1
0.05
0400

500

600

700

800

900 1000

Wavelength (nm)Fig. 5. Estimated relative SRFs.
1.29 nm spectral resolution. The detector originally has the1024⇥1392-pixel size in the spectral and spatial directions;however, a spectral image is recorded using binning to improve the frame rate and signal-to-noise ratio, which resultsin the 512⇥696-pixel size. To simulate HISUI specifications,the spectral sampling distance is downsampled as 10.34 nmwith 64 channels. The MSI is a prototype of HODOYOSHI-1,
which is the Japanese earth-observing micro-satellite. It has a
4-channel line-scanning system with the 4120-pixel-size line
detector. Since the four line detectors are located in parallel
positions on the focal plane as shown in Fig. 3(c), four bands
images have disparities in the scanning direction. We use animage matching technique to register them by shifting.
The two imagers were set up in parallel planes to alignthe cross-track directions and observe a landscape using arotating stage. First, sub-images are extracted and spatiallyco-registered using image matching assuming that captured
objects are relatively far enough from the observation locations comparing with the distance between the centers of
focal planes. The RGB image of multispectral data is shownin the left-side of Fig. 5. The difference of spatial sampling
distance is 6-fold, which is the same ratio of the GSDs ofHISUI between hyperspectral and multispectral imagers. The
two datasets were radiometrically modified and converted
into reflectance using a spectralon target of Labsphere, Inc..
Next, the relative SRFs were estimated using only observed
datasets and the prior knowledge for the spectral ranges. We
used the constrained least squares method, which considers
smoothness and nonnegativity [10]. Fig. 4 shows the estimated relative SRFs of 4 channels. Although we used onlyobserved datasets, the results are consistent with the premeasured SRFs of detectors, which indicates that the onboard
cross-calibration is useful even when the sensor characteristics are unknown or changed on orbit. In the case of HISUI,
the prelaunch SRFs can be used for the constraints to estimate the relative SRFs on orbit [9]. Finally, we applied theCNMF method to the datasets using the relative SRFs andPSF. Fig. 5 shows two zoomed views of the hyperspectral
and fused images in the 483 and 710 nm ranges. As shownin the zoomed images, highly detailed spatial information isacquired for the hyperspectral image.
Since the high-spatial-resolution hyperspectral data is notavailable, the quality of the fused data is evaluated by comparing hyperspectral and multispectral images degraded fromthe fused image with the original input images for evaluatingthat the fused image inherits the information both in the hyperspectral and multispectral images. The spectral angle error(SAE) and the peak signal-to-noise ratio (PSNR) were used to
calculate the spectral and spatial reconstruction qualities, respectively. Table 1 summarizes the average SAE and PSNR.
The evaluation results of the multispectral data is relatively
better than that of the hyperspectral data. The SAE is calculated at each pixel and the PSNR is computed for each band
image; therefore, the spatial information has a large impact
on both criteria. Since the spatial information of the fused
data, i.e., the abundance maps were obtained from the multispectral unmxing, the fused data was biased to the multispectral data in spatial domain, which results in the betterevaluation results with the multispectral data. Errors of thedegraded hyperspectral data are due to the misregistration between the hyperspectral and multispectral images, which is
caused by the difference of viewing angle and optical-systemaberrations and misalignments. It indicates that accurate registration of the two images is essential to obtain high-quality
fused data.
4. AIRBORNE SIMULATION
Airborne simulation tests for the hyperspectral and multispectral data fusion mission on HISUI will be carried on Augustin 2013 and 2014. The study area is the Tama forest sci-RGB image

HSI (483 nm)

Fused (483 nm)

HSI (710 nm)

Fused (710 nm)

Fig. 4. RGB image of multispectral data and two zoomed sub-images of hyperspectral and fused data in 483 and 710 nm.
ence garden in Tokyo, Japan. Hyperspectral and multispectral
imaging sensors are mounted on the same platform to satisfy
the assumption of the same observation conditions. The HSI
is the same imager with the ground experiment. The GSD
is 2.5 m and the swath width is 750 m. The prototype ofHODOYOSHI-1 and EOS 5D Mark II of Canon Inc. willbe mounted as MSIs. EOS 5D Mark II captures 3 spectral
channels, i.e., red, green, and blue, in 14 bit with the 0.25 m
GSD and the 750 m swath width. The hyperspectral sensor
is a pushbrooming imager; however, one of the multispectral sensors is with a 2D image capturing. Therefore, image
registration is significant. HISUI simulation datasets will be
synthesized to evaluate hyperspectral and multispectral data
fusion and to investigate its effectiveness for tree species classification.

5. CONCLUSION
We developed a high-spatial-resolution hyperspectral imaging system combining hyperspectral and multispectral sensors
and presented the ground experiment for the hyperspectral
and multispectral data fusion mission on HISUI. The wholeprocess of data fusion including the preprocessing and crosscalibration was clarified and the feasibility of this mission
was investigated. The preprocessing and cross-calibration areimportant for the final quality of fused data satisfying theassumptions of data fusion. We explained the plan of airborne experiments for further simulation tests. This work
contributes to the practical use of hyperspectral and multispectral data fusion, which can be the prototype trial for thehigher-order product of the HISUI datasets.

7. REFERENCES[1] N. Ohgi, T. Kawashima A. Iwasaki, and H. Inada, “Japanese hypermulti spectral mission,” in Proc. IEEE IGARSS 2011, 2011.
[2] R. C. Hardie, M. T. Eismann, and G. L. Wilson, “Map estimation
for hyperspectral image resolution enhancement using an auxiliary
sensor,” IEEE Transactions on Image Processing, vol. 13, no. 9, pp.
1174–1184, September 2004.
[3] M. T. Eismann and R. C. Hardie, “Application of the stochastic mixing
model to hyperspectral resolution enhancement,” IEEE Transactions
on Geoscience and Remote Sensing, vol. 42, no. 9, pp. 1924–1933,
September 2004.
[4] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,”
IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 2,pp. 528–537, February 2012.
[5] R. Kawakami, J. Wright, Y.-W. Tai, Y. Matsushita, M. Ben-Ezra, andK. Ikeuchi, “High-resolution Hyperspectral Imaging via Matrix Factorization,” Proceedings of the 2011 IEEE Conference on Computer
Vision and Pattern Recognition, pp. 2329–2336, 2011.
[6] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol. 401, pp. 788–791, October
1999.
[7] L. Miao and H. Qi, “Endmember extraction from highly mixed datausing minimum volume constrained nonnegative matrix factorization,” IEEE Trans. Geosci. Remote Sens., vol. 45, no. 3, pp. 765-777,Mar. 2007.
[8] S. Jia and Y. Qian, “Constrained nonnegative matrix factorization for
hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 47,
no. 1, pp. 161-173, Jan. 2009.
[9] N. Yokoya, N. Mayumi, and A. Iwasaki, “Cross-calibration for data
fusion of EO-1/Hyperion and Terra/ASTER,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, inpress, 2013.
[10] G. D. Finlayson, S. Hordley, and P. M. Hubel. “Recovering device

6. ACKNOWLEDGEMENT
This work was supported by contract with the National Institute of Advanced Industrial Science and Technology (AIST)
and KAKENHI (24360347).

sensitivities with quadratic programming,” Proceedings of the IS andT/SID Sixth Color Imaging Conference: Color Science, Systems, andApplications, pp. 90-95, November 1998.