DESIGN OF COMBINED OPTICAL IMAGERS
USING UNMIXING-BASED HYPERSPECTRAL DATA FUSION
Naoto Yokoya1,2 and Akira Iwasaki1
1Research Center for Advanced Science and Technology, The University of Tokyo, Japan
2
Japan Society for the Promotion of Science, Japan
Email: yokoya@sal.rcast.u-tokyo.ac.jp
ABSTRACT

Unmixing-based hyperspectral and multispectral data fusion
enables the production of high-spatial-resolution and hyperspectral imagery with small spectral errors. In this work,
we present sensor design of combined optical imagers using
unmixing-based data fusion, which aims to fuse hyperspectral
and multispectral sensors and improve the performance of thefinal fused data. Owing to the degeneracy of the data cloud
and additive noise, there is an optimal range in the relationship of spatial resolutions between two imagers.
Index Terms— hyperspectral and multispectral data fusion, unmixing, sensor design
1. INTRODUCTION
There are hardware limitations in the sensor design of opticalimagers owing to the trade-offs between spatial resolution,
spectral resolution, swath width, and signal-to-noise ratio(SNR). For instance, with the same number of detectors in
the cross-track direction, if the spatial resolution is enhanced,
the swath width and SNR decrease owing to the limited
number of detectors and the lack of photons, respectively.
Several countries are now developing hyperspectral imaging
cameras as next-generation earth-observing sensors, such as
HISUI, EnMAP, PRISMA, HyspIRI, and HYPXIM [1]–[5].
Generally, to maintain better SNRs than multispectral andpanchromatic sensors, the spatial resolution of hyperspectral sensors is lower than that of conventional imagers. For
example, HISUI, which is a Japanese next-generation earthobserving imager, consists of hyperspectral and multispectral
cameras that have 30 and 5 m ground sampling distances(GSDs), respectively. PRISMA, which is the satellite remotesensing hyperspectral mission developed by Italy, is composed of hyperspectral and panchromatic imagers that have
30 and 5 m GSDs, respectively. Ideally, high resolutions bothin spatial and spectral domains are desirable for optical remote sensing data to obtain better understanding of an object
or phenomenon on Earth.
Unmixing-based data fusion of hyperspectral and otherconventional optical sensors enables the production of high-spatial-resolution and high-spectral-resolution imagery withsmall spectral errors [6]. Combining several sensors using
data fusion software can overcome the limitations of individual hardware. This technology may usher in a breakthroughin the sensor design of satellite optical imagers. Although individual conventional optical sensors were designed to maximize each of their performances according to their mission,
data fusion can be a powerful option for sensor design, whichaims to fuse several sensors and improve the performance ofthe final fused data. Therefore, hyperspectral data obtainedby data fusion expands the possibilities of data specifications
and allows a wide variety of applications. The main functionsof satellite hyperspectral sensors are in the management ofagricultural and forest ecosystems, disaster monitoring, water assessment, and mineral exploration [1]–[5]. The spatialresolutions of spaceborne hyperspectral sensors are approximately 30–60 m GSD, which limits satellite hyperspectral
imaging applications, such as urban land cover classificationof loads and buildings for hazard assessment, and crown-leveltree classification in forests for ecosystem monitoring. Whensatellite hyperspectral imagery can be sharpened to 1–5 mGSD by fusing with multispectral and panchromatic data witha small spectral distortion, it will enable killer applications ofhyperspectral remote sensing. In this work, we demonstrate asensor design of hyperspectral and multispectral imaging system using unmixing-based hyperspectral data fusion by examining the performance of fused data with different sensor
designs.
2. UNMIXING-BASED HYPERSPECTRAL AND
MULTISPECTRAL DATA FUSION
The aim of hyperspectral and multispectral data fusion is toestimate unobservable high-spatial-resolution hyperspectral
data from observable low-spatial-resolution hyperspectral
data and high-spatial-resolution multispectral data, whichhave the trade-off between spectral and spatial resolutions.
We consider the following assumptions for this problem:• Multiple optical imagers are mounted on the same
platform and can obtain images under the same atmo-spheric and illumination conditions.

EndmemberspectraHyperspectral data…

• Observed images are taken over the same areas and geometrically coregistered in preprocessing.

Abundance maps…

Unmixing• Relative sensor characteristics, such as SRF and PSF
are known.

3. EXPERIMENTAL STUDY
We generate synthetic hyperspectral and multispectral datasets
using several combinations of sensor specifications by simulating the trade-off of spatial resolution, spectral resolution,and SNR between hyperspectral and multispectral sensors.
Let us consider a case study of HISUI/VNIR assuming asimple trade-off of sensor specifications. The hyperspectral
and multispectral imagers have 1000 and 18000 pixels in thecross-track direction, respectively. As the default, the spatialresolution was set at 30 and 5 m GSDs for the hyperspectral
and multispectral imagers, respectively, which are based on
the specifications of HISUI. In this case, the swath widths are
30 and 90 km, respectively, and the SNRs are assumed to be
100 considering noisy bands. The multispectral imager has 4spectral channels in the 0.45–0.52-, 0.52–0.60-, 0.63–0.69-,
and 0.76–0.90-µm regions. Here, the simple principle of thespecification trade-off is explained. For example, if the spatial resolution of the hyperspectral sensor is 20 m GSD andRelative spectralresponse function

Relative point
spread function…

Multispectral data

Unmixing…

Unmixing-based hyperspectral and multispectral data
fusion, named coupled nonnegative matrix factorization(CNMF), is composed of alternating unmixing for two images, which is based on a linear spectral mixture model[6]. First, the algorithm starts from the unmixing of the hyperspectral data to estimate endmember spectra taking theadvantage of high spectral resolution. Next, the multispectral data is unmixed after initializing the endmember spectra
and the abundance fractions by using the unmixing resultsof the hyperspectral data. The sequential unmixing for hyperspectral data is processed after initializing the abundance
fractions by using the unmixing results of the multispectral
data. After that, two data are alternately unmixed until convergence and the fused data can be obtained by combining theendmember spectra and the high spatial resolution abundancemaps. Fig. 1 shows the illustration of the algorithm. Whenthe nonnegative matrix factorization (NMF) [7] is used forthe unmixing process, each unmixing converges to a local
minimum, and therefore the initialization is important for theunmixing. Relative spectral response functions are used toinitialize the endmember spectra in the multispectral unmixing and relative point spread functions are used to initialize
the abundance maps in the hyperspectral unmixing. CNMF
uses the advantages of hyperspectral and multispectral data,i.e., spectral and spatial information, respectively, in otherunmixing procedures to find a better local minimum. Moredetails about the CNMF method are given in [6].

High spatial resolution
hyperspectral dataFig. 1. Illustration of unmixing-based hyperspectral and multispectral data fusion
Design 1

Design 2

Design 3

Design 4

30km

60km

90km

10km
90km

20km

Mulspectral

Hyperspectral

Fig. 2. Observation ranges for four sensor designs of hyperspectral (light blue) and multispectral (green) imagers.

the swath width is 20 km, the energy of photons captured bya detector per pixel is the third power of the ratio of spatial
resolution, i.e., 8/27 (= (20/30)3 ) of the energy for the 30-mGSD hyperspectral imager, because the number of incident
photons on a detector at a certain moment is proportional
to the square of the ratio of the spatial resolution and theexposure time is proportional to this ratio. When we simply
assume that the noise is proportional to the square root of
the detected signal intensity, SNR becomes proportional tothe power of 1.5, i.e., 54 (= 100 × (20/30)1.5 ). Therefore,
qualitatively speaking, when the spatial resolution improves,
the observed area decreases and the quality of images deteriorates.
We fix the spatial resolution of the multispectral sensor
and change that of the hyperspectral sensor to 10, 20, 30, and
60 m. In actual designs, HISUI, EnMAP, and PRISMA haveScene 2

Scene 1(a)

4540

44.5

39.5

PSNR (dB)

PSNR (dB)4443.54342.54241.5

39
38.5
38
37.5

4137

40.5
40

1020

30

4050

36.5
1060

20

GSD of HSI(b)

30

40

50

60

50

60

50

60

5060

GSD of HSI32.6

2.8
2.6

SAE (degree)

Fig. 3. CASI-3 images taken over (left: scene 1) forest ofTama, Tokyo in 2009 and (right: scene 2) pasture area ofMotonopporo, Hokkaido in 2008.

SAE (degree)

2.4
2.2
2
1.8

2.4
2.2
2
1.8
1.61.41.61.21.410203040501601020(c)

400

550

350

500
400RMSE

2502001503503002502001001505001001020304050506010203040GSD of HSIGSD of HSI(d)40450300

RMSE

Table 1. Specification of hyperspectral imager.
GSD Swath width SNR
Design 1 10 m
10 km
19
Design 2 20 m
20 km
54
Design 3 30 m
30 km
100
Design 4 60 m
60 km
28330

GSD of HSI

GSD of HSI

243022282026RMSE

30 m GSD and HyspIRI has 60 m GSD. The hyperspectral
sensor specifications are listed in Table 1. Fig. 2 shows thesizes of scenes observed by the two imagers. CNMF is applied to all synthetic datasets and the qualities of fused data
are compared using peak signal-to-noise ratio (PSNR) andspectral angle error (SAE). An airborne hyperspectral dataset
is used to generate the synthetic datasets. Two images werecaptured by CASI-3 with 1 m GSD taken over a forest areain Tokyo in 2009 and a pasture area of Hokkaido in 2008. A
600×600-pixel-size subimage is selected for the simulation,
as shown in Fig. 3. A Gaussian blur filter is used to downgrade the spatial resolution and rectangular SRFs are used togenerate multispectral images. Gaussian noise was added tosatisfy the SNRs of the synthetic datasets. PSNR and SAE
are used to evaluate the final fused products. The number ofendmembers in CNMF is set as 20 for both images.
Fig. 4(a) and (b) show the changes in PSNR and SAE,respectively, with the GSD of the hyperspectral sensor. Whenthe GSDs of the hyperspectral sensor are 20 and 30 m, theperformance of the fused data shows the best results for bothstudy scenes. Since the CNMF fused data is produced bymultiplication of the endmember spectra obtained by hyperspectral unmixing and the abundance maps obtained by multispectral unmixing, the quality of the fused data is determinedby the accuracies of two alternate unmixings. Fig. 4(c) and(d) show the RMSEs of unmixing for the hyperspectral andmultispectral data, respectively. The canges in RMSE areconsistent for the two scenes, and we confirmed that the combination of the accuracies of the two unmixings determines
the performance of the fused data shown in Fig. 4(a) and (b).
To investigate the reason for the changes in the RMSE ofRMSE

1816141224222010188610203040

GSD of HSI5060

1610203040

GSD of HSIFig. 4. (a) PSNR and (b) SAE of fused data, and RMSEsof unmixing for (c) hyperspectral and (d) multispectral data.
Left column is for scene 1 and right column is for scene 2.

unmixing, Fig. 5 shows the data clouds plotted on the space
defined by first and second principal components (PCs) withthe first three endmembers estimated by NMF-based unmixing applied to the hyperspectral data. We compare four hyperspectral datasets with different spatial resolutions, i.e., 5,10, 20, and 30 m GSDs, and the multispectral data with a
5 m GSD. The larger spatial resolution causes more severe
degeneracy. In hyperspectral unmixing, when the number ofendmembers is constant for all datasets, the data cloud withsmaller spatial resolution results in larger unmixing errors
owing to the more complicated structure of the data cloud
and the larger noise. Therefore, the RMSE of unmixing for
hyperspectral data monotonically decreases when the GSDincreases. In multispectral unmixing, endmember spectra areinitialized using those estimated in hyperspectral unmixing.
When the GSD of a hyperspectral sensor increases, the initialized multispectral endmembers become more different fromthe actual endmembers in the 5 m GSD, as shown in the multispectral data cloud. In addition, when the GSD of hyperspectral data decreases, estimated endmembers can contain errors

owing to the severe noise condition of hyperspectral images.
This is the reason why the change in RMSE for multispectral
unmixing results in Fig. 4(d). Since there are opposite trends
in the changes in RMSE for hyperspectral and multispectral
data, there is an optimal range for the GSD of a hyperspectral
sensor, i.e., 20–30 m in this simulation.

spectral camera and fusing them by hyperspectral data fusion.
In this work, we presented a sensor design using combined
optical imagers, which aims to use hyperspectral and multispectral data fusion and maximize the performance of fused
data. We focused on examining the performance of fused
data by changing the relationship of the spatial resolution
between two sensors. The final performance of fused data
is determined by the accuracies of two unmixings. Owing
to the degeneracy of the data cloud and additive noise, there
is an optimal range in the relationship of spatial resolutions
between two imagers. Hyperspectral and multispectral data
fusion based on unmixing contributes to the design of combined optical imagers by presenting a range of optimum
design points. Combined optical imagers can usher in a major breakthrough in specifications of optical remote sensing
sensors and their applications.
5. ACKNOWLEDGMENT
The authors would like to thank Prof. Shuichi Rokugawa forthe many valuable comments and suggestions. N. Yokoya
acknowledges the support from JSPS (Japan Society for thePromotion of Science).
6. REFERENCES[1] N. Ohgi, T. Kawashima A. Iwasaki, and H. Inada,“Japanese hyper-multi spectral mission,” in Proc. IEEEIGARSS, pp. 3756–3759, 2011.
[2] B. Sang, J. Schubert, S. Kaiser, V. Mogulsky, C. Neumann, K.-P. Förster, S. Hofer, T. Stuffler, H. Kaufmann,A. Müller, T. Eversberg, and C. Chlebek, “The EnMAPhyperspectral imaging spectrometer: instrument concept, calibration and technologies,” in Proc. SPIE, vol.
7086, 2008.
[3] C. Galeazzi, A. Sacchetti, A. Cisbani, and G. Babini,“The PRISMA program,” in Proc. IEEE IGARSS, vol.
4, pp. 105–108, 2008.

Fig. 5. From top to bottom, hyperspectral data clouds with10, 20, 30, and 5 m GSDs and multispectral data cloud with 5m GSD plotted on space spanned by 1st (x-axis) and 2nd (yaxis) PCs for Scene 1 (left) and Scene 2 (right). Red, green,and blue circles indicate 3 endmembers estimated from hyperspectral datasets with 10, 20, and 30 m GSDs respectively.

4. CONCLUSION
We can obtain high-spatial-resolution and high-spectralresolution data by observing spectral information using ahyperspectral camera and spatial information using a multi-[4] S. Chien, “Onboard science processing concepts for theHyspIRI mission,” IEEE Intelligent Systems, vol. 24, no.
6, pp. 12–19, 2009.
[5] S. Michel, P. Gamet, and M. J. Lefevre-Fonollosa,“HYPXIM – a hyperspectral satellite defined for science, security and defence users,” in Proc. IEEE WHISPERS, pp. 1–4, 2011.
[6] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral andmultispectral data fusion,” IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 2, pp. 528–537, February 2012.
[7] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol.
401, pp. 788–791, October 1999.