EFFECT OF UNMIXING-BASED HYPERSPECTRAL SUPER-RESOLUTIONON TARGET DETECTION
Naoto Yokoya and Akira Iwasaki
Department of Advanced Interdisciplinary Studies, The University of Tokyo, Japan
ABSTRACT
We present an airborne experiment on unmixing-based hyperspectral super-resolution using RGB imagery to examine therestoration of pure spectra comparing with ground-measured
spectra and demonstrate its impact on target detection. An
extended version of coupled nonnegative matrix factorization(CNMF) is used for hyperspectral super-resolution to deal
with a challenging problem setting. Our experiment showedthat the extended CNMF can restore pure spectra, which contribute to accurate target detection.
Index Terms— Hyperspectral super-resolution, data fusion, unmixing, coupled nonnegative matrix factorization(CNMF)
1. INTRODUCTION
Hyperspectral (HS) imagers generally have a larger groundsampling distance (GSD) than multispectral (MS) imagers
owing to a trade-off of sensor design between spatial andspectral resolutions and the signal-to-noise ratio. HS and MS
data fusion enables super-resolution of HS data [1, 2, 3]. A
Bayesian approach was first proposed for HS super-resolution
using MS data [1, 2]. Unmixing-based HS and MS data fusion can enhance the spatial resolution of HS data with little
spectral distortion [3]. An unmixing-based HS and MS data
fusion method, named coupled nonnegative matrix factorization (CNMF), was proposed for remote sensing [3]. CNMF
is composed of alternating unmixing for two images using
nonnegative matrix factorization (NMF) [4].
HS and MS data can be represented in matrix form as X ∈RLh ×Ph and Y ∈ RLm ×Pm , respectively. Lh and Lm denotethe numbers of spectral bands, and Ph and Pm denote thenumbers of pixels. The high-spatial-resolution HS data to beestimated is denoted as Z ∈ RLh ×Pm . The spectrum at eachpixel z ∈ RLh ×1 is assumed to be a linear combination ofseveral endmember spectra. Therefore, Z is formulated as
Z = EA + N,(1)where E ∈ RLh ×M is the endmember matrix and M being
the number of endmembers. A ∈ RM ×Pm is the abundance
matrix and N ∈ RLh ×Pm is the residual. Unmixing-based HSand MS data fusion yields estimates of E and A from observable X and Y to reconstruct Z. The low-spatial-resolution
HS data and MS data are modeled as
X = ZS,Y = RZ.

(2)

Here, S ∈ RPm ×Ph is the spatial response transform matrix
and R ∈ RLm ×Lh is the spectral response transform matrix, which are defined as relative sensor characteristics inthe spatial and spectral domains, respectively. By substituting (1) into (2) and (3), X and Y can be approximated as twoLSMMs:
X ≈ E(AS),Y ≈ (RE)A.

(3)

E and A can be obtained by alternating unmixing of X andY under the constraints of the relative sensor characteristics(R and S). Here, we define Ah = AS and Em = RE
as degraded versions of A and E, respectively. Spatial andspectral consistency, which is expressed as RX = YS from(2) and (3), is important in practical use.
HS and MS data fusion algorithms in remote sensing havemainly been evaluated using synthetic datasets or real datasets
taken from different platforms [5]. In this work, we present anairborne experiment on unmixing-based hyperspectral superresolution to examine the restoration of pure spectra using
ground measurements and show its impact on target detection. An extended CNMF is introduced to deal with a challenging problem setting: only three channels for MS data anda 10-fold GSD difference.
2. MATERIALS AND METHODS
2.1. Image Acquisition and Preprocessing
Airborne observation was conducted using HS and RGB
cameras mounted on a small aircraft. The HS camera is aHyperSpec-VNIR-C (Headwall Photonics Inc.), which captures 128 bands in the 390–1040 nm spectral range by pushbroom imaging. The 400–800 nm spectral region was usedbecause the remaining wavelength ranges have low accuracyof reflectance conversion. The RGB camera is an EOS 5D
Mark II (Canon Inc.). The altitude was approximately 1000m and the GSDs of the HS and RGB images are 2.5 andGreen sheets

GravelFig. 1. Color images of study area obtained by (left) HS and (right) RGB cameras.

Low spatial resolution

Interpolated

SegmentedSegmented+BlurredFig. 2. Improved initialization of high-spatial-resolution abundance map.
0.25 m, respectively, after geometric projection. The dataset
was acquired over the University of Tokyo campus and theneighboring urban area. Fig. 1 shows color images of part ofthe study area obtained using HS and RGB cameras. We set
two green plastic sheets as targets to be detected on a roof ofthe building where is covered by gravel.

To ensure spatial and spectral consistency between HS
and RGB images, the preprocessing is important: (i) Smileand keystone are modified using an image-matching technique[6]. (ii) Geometric projection is applied to two datasets andthe HS image is registered to the RGB image to mitigate geometric errors caused by the difference in scanning type, i.e.,line and area scanning. The relative spatial response function (S) can be approximated by a Gaussian filter. (iii) TheHS image is converted to a reflectance image using groundmeasured spectra. A bright homogeneous walkway was usedas the reference area. (iv) The estimation of relative spectral response functions (SRFs) can be formulated as a constrained least-squares problem [?]. Reflectance conversion
coefficients of the RGB image are simultaneously obtained.
(v) The RGB image is converted to reflectance using the coefficients.

2.2. Extended CNMF
After the preprocessing, the high-spatial-resolution HS data
is obtained by extended CNMF. Since the difference in GSDs
is 10-fold and only three spectral channels are available forhigher-spatial-resolution information, the unmixing of theRGB image is a severely ill-posed problem. To tackle thischallenging problem, we improve the initialization of theabundance matrix (A) in the unmixing of the RGB image.
CNMF starts with NMF-based unmixing of the HS data
to estimate the endmember (E) and abundance (Ah ) matrices
using its spectral advantage. NMF converges to local minima;
therefore, the initialization is important. The vertex component analysis (VCA) [8] and fully constrained least-squares(FCLS) [9] methods are used for the initialization of E andAh , respectively.
Next, the RGB image is unmixed by NMF after initializing the endmember (Em ) and abundance (A) matrices using
E and Ah . The relative SRFs (R) are used for initializing
Em . In the ordinary CNMF, A is initialized as interpolated
abundance maps calculated from Ah , which is not suitable forthe 10-fold GSD difference. In this work, we introduce segmentation for better initialization of A, as shown in Fig. 2.
The interpolated abundance maps are converted to segmented
abundance maps by integrating region-unifying segmentationof the RGB image with the interpolated abundance maps. TheAAB

BCAB

Plastic sheet

CCFig. 3. 719 nm band images of (left) HS and (right) fused data. Subimages A and C show tree crowns and B shows the plasticsheet.
average abundance value of a segment is assigned to all pixels in the segment. Finally, a blurred version of segmented
abundances is used for initializing A. In addition, we update
only A in RGB unmixing to avoid the incorrect convergenceof Em owing to the small number of spectral bands.
The sequential unmixing for HS data is processed after
initializing the abundance by Ah = AS. After that, twodatasets are alternately unmixed until convergence and thefused data is obtained by multiplying the endmember matrix
by the high-spatial-resolution abundance matrix.

2.3. Target Detection
To examine the effectiveness of unmixing-based HS superresolution for practical applications, we use the fused data
for target detection. Our target is the green plastic sheet andits spectrum is obtained by ground measurements using an
USB2000+VIS-NIR (Ocean Optics Inc.). We adopt a subpixel target detection method based on linear spectral unmixing named nonnegatively constrained least squares (NCLS)[10]: 1) The target spectrum is set as the first endmember
E = [e0 ]. Let k = 0 and select ϵ to be a prescribed error
threshold. 2) Let k ← k + 1 and the abundance fractions
are estimated with the endmember matrix E = [e0 ...ek−1 ] bysolving the NCLS problem:mina1∥z − Ea∥22 subject to a ≽ 0,2(4)3) Check the least squares error if ∥z − Ea∥22 < ϵ for all z. If
it is, the algorithm stops, otherwise continue. 4) Find a newendmember as
ek = argmax ∥z − Ea∥22z(5)3. EXPERIMENTAL RESULTS
We produced the 0.25-m-GSD HS data with 97 bands over the
400–800 nm spectral region by applying the extended CNMF
to the HS and RGB images. Fig. 3 shows 719-nm-band images of low-spatial-resolution HS and fused data. As shownin the enlarged images in Fig. 3, tree crowns and the green
plastic sheet spread on the roof are identifiable with the fused
data owing to the 10-fold improvement of GSD.
To evaluate the spectral reconstruction of the fused data,
we examine the spectral profile of the green plastic sheet
shown in Fig. 3. The left graph in Fig. 4 shows the spectraof the green plastic sheet for low-spatial-resolution HS andfor CNMF and extended-CNMF data as well as the groundmeasured spectra of the green plastic sheet and the surrounding gravel. The spectrum of the low-spatial-resolution HS
data appears to be a mixture of the two ground-measured
spectra because of mixed pixels. In contrast, the peak aroundthe green region is restored for the extended-CNMF data,
whereas the CNMF spectrum contains artificial errors. This
result indicates that the segmentation-integrated initializationof higher-spatial-resolution abundance maps contributes to abetter convergence of the RGB-image unmixing, which enables the restoration of pure spectra. The difference between
the extended-CNMF and ground-measured spectra may becaused by errors in the preprocessing or in the ground measurements.
Next, we applied the NCLS subpixel target detection toHS, MS, CNMF, and extended-CNMF datasets and the targetabundances are shown at the right images of Fig. 4. The abundances at the green plastic sheets obtained by the extendedCNMF data show higher values compared to those of theother data. The clearer abundance contrast between the target objects and backgrounds is expected to contribute to better target detection. Surprisingly, the abundances of the targetobjects obtained by the CNMF data show lower values com-0.35HSReflectance

MSCNMF

0.64

Extended CNMF

Green sheet
Ground truth
Extended CNMF
CNMF
HS data

0.3
0.25
0.20.15(a) 0.276(b) 0.340

0.1
0.05
0400

(a) 0.519(b) 0.587(a) 0.274(b) 0.351(a) 0.600(b) 0.643

GravelGround truth
450500

550

600

650

700

750

800

Wavelength (nm)0Fig. 4. (Left) Spectral validation of fused data using ground-measured spectra. Right images show abundance maps of green
plastic sheet for HS, MS, CNMF, and extended CNMF data with two peak values.
pared to the HS and MS data because of spectral errors. It suggests that the careful preprocessing and the extended-CNMF
are effective in practical use with challenging optimization
conditions.
4. CONCLUSION
In this work, we presented an airborne experiment of unmixingbased HS super-resolution using an RGB image to examine
the restoration of pure spectra and its effect on target detection. We introduced segmentation to the initialization ofhigher-spatial-resolution abundance maps in CNMF to deal
with a severe ill-posed problem due to the 10-fold GSD difference and lack of higher-spatial-resolution near-infrared
image in the MS data. The extended CNMF method not only
improved the spatial resolution but also restored pure spectra,
which contributes to accurate target detection.
5. REFERENCES[1] R. C. Hardie, M. T. Eismann, and G. L. Wilson, “Mapestimation for hyperspectral image resolution enhancement using an auxiliary sensor,” IEEE Trans. Image Processing, vol. 13, no. 9, pp. 1174–1184, 2004.
[2] M. T. Eismann and R. C. Hardie, “Application of thestochastic mixing model to hyperspectral resolution enhancement,” IEEE Trans. Geosci. Remote Sens., vol. 42,no. 9, pp. 1924–1933, 2004.
[3] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral andmultispectral data fusion,” IEEE Trans. Geosci. RemoteSens., vol. 50, no. 2, pp. 528–537, 2012.

[4] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol.
401, pp. 788–791, 1999.
[5] N. Yokoya, N. Mayumi, and A. Iwasaki, “Crosscalibration for data fusion of EO-1/Hyperion andTerra/ASTER,” IEEE J. Sel. Topics Appl. Earth Observ.
Remote Sens., vol. 6, no. 2, pp. 419–426, 2013.
[6] N. Yokoya, N. Miyamura, and A. Iwasaki, “Detectionand correction of spectral and spatial misregistrations
for hyperspectral data using phase correlation method,”
Applied Optics, vol. 49, no. 24, pp. 4568–4575, 2010.
[7] G. D. Finlayson, S. Hordley, and P. M. Hubel. “Recovering device sensitivities with quadratic programming,” inProc. the IS and T/SID Sixth Color Imaging Conference:Color Science, Systems, and Applications, pp. 90–95,
1998.
[8] J. M. P. Nascimento and J. M. B. Dias, “Vertex component analysis: A fast algorithm to unmix hyperspectral
data,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 4,pp. 898–910, 2005.
[9] D. C. Heinz and C.-I. Chang, “Fully constrained leastsquares linear spectral mixture analysis method for material quantification in hyperspectral imagery,” IEEE
Trans. Geosci. Remote Sens., vol. 39, no. 3, pp. 529–545, 2001.
[10] C.-I. Chang and D. C. Heinz, “Constrained sub pixel
target detection for remotely sensed imagery,” vol. 38,no. 3, pp. 1144–1159, 2000.