IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

1Hyperspectral and Multispectral Data Fusion:
A Comparative Review
Naoto Yokoya, Member, IEEE, Claas Grohnfeldt, Student Member, IEEE, and Jocelyn Chanussot, Fellow, IEEE

Abstract—In recent years, enormous efforts have been madeto design image processing algorithms to enhance the spatial
resolution of hyperspectral (HS) imagery. One of the most commonly addressed problems is the fusion of HS data with higherspatial-resolution multispectral (MS) data. Various techniques
have been proposed to solve this data fusion problem based on
different theories including component substitution, multiresolution analysis, spectral unmixing, and Bayesian probability. This
paper presents a comparative review of those HS-MS fusion
techniques with extensive experiments. Ten state-of-the-art HSMS fusion methods are compared by assessing their fusion
performance both quantitatively and visually. Eight data setsfeaturing different geographical and sensor characteristics areused in the experiments to evaluate the generalizability andversatility of the fusion algorithms. To maximize the fairness andtransparency of this comparison, publicly available source codes
are used, and parameters are individually tuned for maximum
performance. Additionally, the impact of spatial resolution enhancement on classification is investigated. Robustness against
various factors characterizing the HS-MS fusion problem issystematically analyzed for all methods under comparison. The
algorithm characteristics are summarized, and methods with highgeneral versatility are clarified. The paper also provides possible
future directions for the development of HS-MS fusion.
Index Terms—Hyperspecral and multispectral data fusion,
resolution enhancement, comparative review.

U

I. I NTRODUCTION

pcoming spaceborne imaging spectroscopy (or hyperspectral) missions [1]–[6] will enable the identificationand discrimination of materials and the derivation of surface
parameters at an accuracy level unattainable by currently operational optical broadband (multispectral) satellites. Those missions are designed and expected to enhance our understanding,possibilities, and capabilities in a wide range of applications
such as monitoring and management of natural resources, theecosystem, biodiversity, and disasters. Due to the inevitable
This work was supported by the Kayamori Foundation of Information
Science Advancement, Japan Society for the Promotion of Science (JSPS)
KAKENHI 15K20955, and Alexander von Humboldt Fellowship for postdoctoral researchers.
N. Yokoya is with the Department of Advanced Interdisciplinary Studies,
the University of Tokyo, Tokyo 153-8904, Tokyo, Japan, with the Remote
Sensing Technology Institute (IMF), German Aerospace Center (DLR), 82234
Wessling, Germany, and also with Signal Processing in Earth Observation(SiPEO), Technical University of Munich (TUM), 80333 Munich, Germany(e-mail: yokoya@sal.rcast.u-tokyo.ac.jp).
C. Grohnfeldt is with the Remote Sensing Technology Institute (IMF),
German Aerospace Center (DLR), 82234 Wessling, Germany (e-mail:claas.grohnfeldt@dlr.de).
J. Chanussot is with the Grenoble Images Parole Signal Automatique Laboratory, Department of Image and Signal, Grenoble Institute of Technology,
Saint-Martin-d’Hères F-38402, France, and also with the Faculty of Electricaland Computer Engineering, University of Iceland, Reykjavik 101, Iceland (email: jocelyn.chanussot@gipsa-lab.fr).

trade-off between spatial resolution, spectral resolution, andsignal-to-noise ratio (SNR), spaceborne imaging spectrometers
are usually designed to provide data with a moderate groundsampling distance (GSD) (e.g., 30 m; see the illustration
at the upper left of Fig. 1), limiting the range of potential
applications.
If higher spatial resolution – possibly multispectral (MS) –data of the scene of interest is available, data fusion can be performed to generate high spatial resolution hyperspectral (HS)data (see Fig. 1). This data can be thought of as the product ofa synthetic sensor featuring the high spatial resolution of the
MS sensor and the high-spectral resolution of the HS sensor.
It allows for various new applications potentially conducted
on a global scale, which, to data, have been possible only
locally with high-spatial-resolution airborne imaging systems.
Such applications include high-spatial-resolution ecosystemmonitoring, or the high-spatial-resolution mapping of e.g.
minerals, urban surface materials, plant species, and manyothers. Although the number of available satellite platforms
mounting both HS-MS imaging sensors is limited to date[6], the increasing number and availability of high-resolution
optical satellites [7]–[9] as well as the ever improving revisit
cycles allow for acquisitions of complementary HS-MS images
during the same season and possibly under similar atmosphericand illumination conditions. Fig. 2 shows a scatter plot of
currently operational MS satellite missions and both operating
and upcoming HS satellites over GSD (y-axis) and the numberof spectral bands between 0.4–2.5 µm (x-axis). Considering
the currently – or soon to be – available satellite pair constellations, the potential synthesized high-resolution HS sensors
would fall within the purple area in the lower right corner in
Fig. 2.
A variety of HS-MS data fusion techniques have beendeveloped in the last decade to enhance the spatial resolutionof HS imagery as detailed in Section II. Because most investigators used limited data sets with slightly different evaluation
methodologies, the generalizability and versatility of various
HS-MS fusion methods remain unknown. Loncan et al. presented a comparative study of hyperspectral pan-sharpening,which is a special case of HS-MS fusion [10]. Mookambiga
and Gomathi [11] and Palubinskas [12] reported reviews ofresolution enhancement of HS data including HS-MS fusion;
however, no comparative experiment was provided. To thebest of the authors’ knowledge, there is no study comparing
different state-of-the-art HS-MS fusion methods with extensive
experiments sufficient to draw meaningful conclusions.
In this paper, we compare ten state-of-the-art HS-MS fusion
methods on a variety of data sets of different nature andIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

2Hyperspectral Data

Multispectral Data

30 m
GSD

Applications:
Mineral mapping, identification of plant species, etc.

5m
GSD

Applications
Land cover mapping, object recognition, etc.

0Abundance

N

1 km

0

0.6
AluniteKaolinite
Muscovite

N

30 m

30 m

50 m

5m

Wavelength

Mixed pixel(tree+soil+road)0Ŷ Orchard grassŶ Reed canary grassŶ Kentucky bluegrassŶ Sweet vernal grassŶ Elytrigia repens

400 m

Wavelength0

Pure pixel (tree)

0

NDVI

1
500 m

Hyperspectral and Multispectral Data Fusion

Fused Data
5m
GSD

Applications:
High-resolution mapping of urban surface materials, minerals, plant species, etc.

5m

Wavelength

Pure pixel (tree)Ŷ Healthy grassŶ Stressed grass
Ŷ Trees
Ŷ SoilŶ WaterŶ Residential
Ŷ Commercial
Ŷ RoadŶ HighwayŶ Railway
Ŷ Parking lot 1
Ŷ Parking lot 2

American sweetgum
Japanese bigleaf magnolia
Painted maple
Oriental raisin tree
Chinese evergreen oak
Japanese blue oak
Konara oak

35°65' N

California incense cedar
Bald cypress
Japanese cypress
Japanese cedar
Deodar cedar
Loblolly pine
Eastern white pine
Koyama’s spruce
Momi fir

N
0

200 m

139°28' E

Fig. 1. Concept and benefits of HS-MS data fusion. Combining the high spatial resolution information and corresponding data analysis methods in MS
data with the high spectral resolution information and corresponding material identification and discrimination capabilities opens whole new categories ofapplications involving high spatial high spectral resolution data possibly on a global scale.

102

Sentinel-2 (60 m)

HyspIRI
CHRIS

Landsat-8

GSD (m)Sentinel-2 (20 m)
10

1

HISUI Hyperion
DESIS
PRISMA
EnMAP

CHRIS

Sentinel-2 (10 m)

ShalomHYPXIM-P

RapidEye
WV-3-SWIR

Target of
HS-MS fuion

QuickBird

10

WV-2
WV-3-VNIR

0

10

0

1

10
Number of bands

102Fig. 2. GSD and the number of spectral bands in 0.4–2.5 µm of spaceborne
MS (red dot) and HS (blue dot) sensors.

characteristic. The main contributions of this paper are thefollowing:•

Objective and fair comparison: To conduct an objective
and fair comparison, a total of eight data sets withdiversity in scenes and fusion scenarios including but notlimited to various resolution ratios and spectral overlap
between the two input images were used. Publicly available source codes are used for the experiments. All methods are optimized for maximum individual performancein every experiment by extensive parameter tuning andoptimal algorithm setting.
• Comprehensive evaluation methodology: Both quantitative and visual assessment of all fusion results are conducted. A complementary selection of well-establishedevaluation metrics ensure an objective comparison of theresolution enhancement power of all investigated algorithms and reveal individual drawbacks and advantages
relative to the other methods. Furthermore, applicationdriven evaluation of the fused data is performed byexamining the impact of all fusion results on pixel-wiseclassification tasks.
• Analysis of algorithm characteristics: Characteristics,
strengths, and drawbacks are identified and discussed forboth individual and categories of the ten state-of-the-art
HS-MS fusion algorithms with different fusion scenarios.
On the basis of the analyzed algorithm characteristics, we
present potential future directions for the development of
HS-MS fusion.
The remainder of the paper is organized as follows. Section II is devoted to an overview of the development in HSMS fusion. Section III briefly outlines the ten state-of-the-art
methods for HS-MS fusion under investigation. Section IV
presents the eight data sets used in the comparative study
and the evaluation methodology. Experimental results anddiscussions are provided in Section V. Section VI concludes
the paper.

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

II. D EVELOPMENT IN H YPERSPECTRAL ANDM ULTISPECTRAL DATA F USION
Pan-sharpening has been widely developed and used toenhance the spatial resolution of MS imagery by fusing the
MS data with a corresponding higher-resolution panchromatic
image [13]–[18]. A large number of pan-sharpening techniques
have been developed over the past two decades; representativetechniques can be roughly categorized as follows: 1) component substitution (CS) [19]–[21], 2) multiresolution analysis(MRA) [22], [23], and 3) sparse representation (SR) [18],[24], [25]. As pan-sharpening can be considered a special
case of the HS-MS fusion problem, efforts have been madetowards generalizing existing pan-sharpening methods for HSMS fusion. One of the first attempts of pan-sharpening-based
HS-MS fusion was presented in [26], [27] using a wavelet
technique. However, the performance was highly dependent
on the spectral re-sampling method, which caused difficulties
in enhancing the spatial resolution of all HS band images.
Based on recent advances in pan-sharpening, more sophisticated attempts have been made to adapt pan-sharpening
techniques to the general HS-MS fusion problem. Chen etal. proposed a framework that solves the HS-MS fusion
problem by dividing the spectrum of HS data into several
regions and fusing HS and MS band images in each region
by conventional pan-sharpening techniques [28]. A synthetic
image is generated by spectral re-sampling of the MS data andused as a high-resolution image in the spectral range that is
not covered by MS bands. In [29], [30], a recent SR-based
pan-sharpening method [18] was applied to HS-MS fusion,
before the authors demonstrated the applicability of their spectral grouping concept to any pan-sharpening technique [31].
Selva et al. proposed a framework called hypersharpeningthat effectively adapts MRA-based pan-sharpening methods
to HS-MS fusion by synthesizing a high-resolution image foreach HS band as a linear combination of MS band images
via linear regression [32]. It was shown that the synthesizedhigh-resolution bands used in hypersharpening could lead tosignificantly better fusion results than a similar yet simpler
approach in which for each HS band one high-resolution MS– namely the most correlated – band is selected from theavailable MS bands rather than synthesized.
Another popular approach fuses the HS-MS images by
exploiting the inherent spectral characteristics of the scene
via a subspace spanned by a set of basis vectors or spectral
signatures of underlying materials (so-called endmembers). A
Bayesian method based on a maximum a posteriori (MAP)
estimation developed by Eismann et al., one of the first methods published in the technical literature, was a breakthroughin terms of enhancing the spatial resolution of all HS band
images using a higher-resolution data source acquired by apanchromatic or MS imager [33]–[36]. This method uses astochastic mixing model (SMM) to estimate the underlying
spectral scene characteristics and formulates a cost functionthat optimizes the estimated HS data relative to the input HSMS images. It should be noted that the optimization of MAPSMM was processed in the principal component subspace.
This idea of fusing the HS-MS images based on spectral

3

information of both input images on a subspace has been themain source of inspiration for many HS-MS fusion methods
developed later [37]–[39].
The principles of spectral unmixing have been used in multisensor multiresolution image fusion already around 2000 [40]–[42]. Gross and Schott proposed a method that unmixes lowresolution HS data and sharpens the abundance maps by fusing
it with higher-resolution panchromatic data using constrained
optimization techniques [40], [41]. Zhukov et al. presented
a technique that unmixes low-resolution images using spatial
details of segmentation obtained from high-resolution images
and finally sharpens low-resolution images by assigning theestimated endmember signatures to the corresponding highresolution pixels of the segmentation map [42]. In recent years,
the idea of using spectral unmixing in HS-MS fusion has been
attracting considerable attention owing to its straightforward
interpretation of the fusion process. Several unmixing-based
methods have been proposed for HS-MS fusion, resulting invarious optimization formulations which have led to stateof-the-art fusion performance [43]–[55]. The unmixing-based
fusion idea aims at obtaining endmember information andhigh-resolution abundance matrices from the HS-MS images,
respectively, under the constraints of relative sensor characteristics, such as a spectral response function (SRF) and a point
spread function (PSF). The fused image can be reconstructed
as the product of the two resulting matrices.
Berné et al. presented an HS-MS fusion method basedon nonnegative matrix factorization (NMF) for mid-infrared
astronomy [43]. The low-resolution HS image is unmixed
by NMF, and the high-resolution abundance maps are obtained from the MS image by least squares regression; theresampled endmember signatures are fixed. Yokoya et al.
proposed coupled nonnegative matrix factorization (CNMF)
that estimates the endmember and abundance matrices via
alternating spectral unmixing based on NMF under the constraints of an observation model which incorporates both therelative SRF and PSF [44], [48]. Kawakami et al. [46] andWycoff et al. [49] used sparse regularization on the fractional
abundances, assuming that the number of endmembers at
each pixel is small compared to the number of available
endmembers in the underlying endmember matrix. Akhtar
et al. [50] proposed a method based on dictionary learningand sparse coding to obtain the endmember and abundance
matrices, respectively. Lanaras et al. [54] introduced a projected gradient method into the alternate updates of the twounderlying matrices. Veganzones et al. [56] demonstrated that
local image processing of the unmixing-based approach has
potential to mitigate the ill-posedness of the fusion problem
and, therefore, improve the quality of the fusion result.
Wei et al. developed a Bayesian HS-MS fusion methodology
using both a subspace transformation and a regularization in
the fusion problem to cope with the ill-posed inverse problem.
The problem formulation is based on information of the priordistribution in the observed scene such as Gaussian or sparsity
promoted Gaussian [38], [57], [58]. Similar to MAP-SMM,
the optimization problem was formulated on a subspace corresponding to the principal components of the input HS data.
Recently, a Sylvester equation-based explicit solution was in-IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

tegrated into the Bayesian HS-MS fusion methodology, named
the fast fusion based on Sylvester equation (FUSE) [59]. FUSE
significantly decreased the computational complexity whileachieving the same performance as the previous Bayesian HSMS fusion algorithms. Simões et al. proposed a method called
HySure based on vector-total-variation-based regularization of
the spatial distribution of subspace coefficients, where the subspace can be defined either by singular value decomposition(SVD) or by endmember spectral signatures [39].

4

TABLE IP ROPERTIES OF HS-MS FUSION METHODS : T YPE OF SUBSPACE
TRANSFORMATION AND INDICATION OF THE DEPENDENCY ON THE
KNOWLEDGE OF SRF AND PSF (Y ES /N O ).

MethodGSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

III. S ELECTED HYPERSPECTRAL AND MULTISPECTRALDATA FUSION METHODSIn this comparative study, ten HS-MS fusion methods are
selected to represent the state-of-the-art in HS-MS fusion
including established methods as well as recently publishedones. All methods under comparison correspond to at least
one of the following four categories: 1) CS, 2) MRA, 3) unmixing, and 4) Bayesian-based approaches. The unmixing andBayesian-based methods fall into the same broader category ofsubspace-based methods. Fig. 3(a) depicts those categories andthe correspondence with the investigated methods. Flowcharts
describing the fusion processes of the CS, MRA (hypersharpening), and subspace-based approaches are illustrated in
Figs. 3(b)–(d), respectively. The properties of the ten methods
are summarized in Table I. Gram-Schmidt adaptive (GSA) [21]is adopted as the representative CS-based pan-sharpening
method. Two MRA-based pan-sharpening methods, namelysmoothing filtered-based intensity modulation (SFIM) [22] andgeneralized Laplacian pyramid (GLP) [23], are adapted to HSMS fusion via hypersharpening. CNMF [48], HySure [39],Akhtar’s [50], and Lanaras’s [54] methods represent theunmixing-based approach (Akhtar’s and Lanaras’s methods
are referred to as ECCV’14 and ICCV’15, respectively).
MAP-SMM [36] and two versions of FUSE [59] are basedon Bayesian probability. Appart from the Bayesian-based
methods, which use principal component analysis (PCA) for
subspace transformation, also the unmixing-based approaches
can be considered as subspace methods, because the dimensionof the space spanned by the endmembers in these methods
is generally smaller than the dimension spanned by the unknown high resolution HS image. All ten methods are briefly
described below.
A. GSA
Aiazzi et al. improved CS pan-sharpening methods bytaking into account the influence of SRF on the fusion
procedure [21]. In the general CS scheme, a low-resolution
image is sharpened by adding spatial details obtained by multiplying the difference between a high-resolution image anda synthetic intensity component by a band-wise modulation
coefficient. The improvement lies in computing the synthetic
intensity component by performing a linear regression between
a high-resolution image and lower-resolution bands to mitigate
spectral distortion. GSA integrates this technique into the GS
algorithm [20]. The most straightforward way to apply GSA to
HS-MS data fusion is to construct multiple image sets for pansharpening subproblems where each set is composed of one

Subspace
GS
–
–
VCA
Spams
SISAL
VCA / SVD
PCA
PCA
PCA

SRF
No
No
No
Yes
Yes
Yes
Yes
No
Yes
Yes

PSF
No
No
Yes
Yes
No
Yes
Yes
No
Yes
Yes

MS band and corresponding HS bands grouped by correlationbased clustering.
B. SFIM-HS
Based on a simplified model for solar radiation and land
surface reflection, SFIM sharpens the low resolution image
by multiplying an upscaled lower resolution image by a ratio
between a higher resolution image and its low-pass filteredversion on a pixel-by-pixel basis [22]. SFIM can be performed
on individual HS bands. A high resolution image can beeither a selected MS band based on correlation analysis ora synthesized image obtained by a linear regression of MS
bands via least squares methods, while the latter is referred toas hypersharpening in [32]. The hypersharpening technique isadopted in this work. The SFIM hypersharpening method isreferred to as SFIM-HS.
C. GLP-HS
In the GLP fusion scheme [23], spatial details of each
low-resolution band are obtained as the difference between
a high-resolution image and its low-pass version multiplied
by a gain factor, which can be computed either locally or
globally. In this paper, a global gain as given in [32] isadopted. A Gaussian filter, matching the modulation transfer
function (MTF) of a lower-resolution sensor, is used for lowpass filtering. As with SFIM, hypersharpening is used here toeffectively adapt the GLP fusion scheme to HS-MS fusion.
The GLP hypersharpening method is referred to as GLP-HS.
D. CNMF
CNMF [48] alternately unmixes the HS-MS images by
NMF [60] to estimate the spectral signatures of endmembers
and the high resolution abundance maps, respectively. CNMF
starts by unmixing the HS image using vertex component analysis (VCA) [61] to initialize the endmember signatures. Sensor
observation models that relate the two input images with therelative sensor characteristics (i.e., SRF and PSF) are built into
the initialization of the MS signatures of endmembers and the
low-resolution abundance maps to find better local optima.
The final high-resolution HS data is obtained as the productof the spectral signatures and the high-resolution abundancemaps.

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

5

HS dataInterpolated HS bands

CS

Pansharpening-based methods
MRA

GSA

Low-resolution MS data

SFIM-HS GLP-HS

Repeat for all
MS bands

Coefficients
Spectragrouping

Subspace methods
Unmixing

High-resolution
HS bands

SelectHS bands

HS bands

Synthesized band

Bayesian

CNMF ICCV15

HySure

MAP-SMM

Coefficients

FUSE / FUSE-S
Histogrammatching

Select one
MS band

ECCV14

MS data

MS band(a) Categories+ -Detail image

High-resolution HS data
Addition
Multiplication
Low-pass filtering
Downsampling
Upsampling(b) CS-based approach

HS data
HS bandHigh-resolution
HS band

Interpolated HS band

Select one
HS bandHS basisLow-resolution MS data

Repeat for all
HS bands

Calculatescaling
Linearregression

Low-resolution
coefficients

Subspace
representation

HS data

Detail image

MS data+-

Synthesized band

High-resolution HS data

Blurred synthesized band

Addition
Multiplication
Low-pass filtering
Downsampling
Upsampling(c) MRA-based approach (hypersharpening)

Subspace
representation

MS data
High-resolution HS data

MS basisHigh-resolution
coefficients

Multiplication
Low-pass filtering
Band-pass filtering
Downsampling
Upsampling(d) Subspace-based approach

Fig. 3. (a) Categories of HS-MS fusion methods and correspondences to the specific methods under comparison. Flowcharts of (b) CS, (c) MRA(hypersharpening), and (d) subspace-based approaches.

E. ECCV’14
Akhtar et al. applied dictionary learning and sparse coding
to obtain the endmembers and the high-resolution abundances,respectively [50]. The spectral signatures of endmembers are
obtained by the online dictionary learning method proposed by
Mairal et al. [62]. A simultaneous greedy pursuit algorithm,
named G-SOMP+, was proposed to learn a sparse code, whichcorresponds to the high-resolution abundances. G-SOMP+ isprocessed with respect to small disjoint spatial patches. The
algorithm requires the availability of the relative SRF.
F. ICCV’15
Similar to CNMF, Lanaras’s algorithm [54] jointly unmixes
the two input images into the spectral signatures of endmembers and the associated fractional abundances. A projected gradient method was proposed to alternately update
the endmember signatures and the high-resolution abundancesby solving the two unmixing problems of the input HS-MS
images, respectively. Simplex identification via split augmented
Lagrangian (SISAL) [63] is used to initialize the endmembers
and sparse unmixing by variable splitting and augmented
Lagrangian (SUnSAL) [64] is adopted to obtain initial abundances. The availability of both spatial and spectral relative
sensor characteristics is required to initialize the degradedversions of the endmembers and abundances.
G. HySureHySure introduced total variation regularization into thesubspace-based HS-MS fusion framework [39]. The total
variation regularization is effective in preserving edges whilesmoothing out noise in homogeneous regions. The fusion problem is formulated as the minimization of a convex objectivefunction with respect to subspace coefficients, which can beseen under a Bayesian approach. The subspace transformation
is derived from the low-resolution HS image either by anIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

endmember extraction technique (e.g., VCA) or by SVD,
where the former version falls into the unmixing scheme. In
this paper, we adopt the unmixing version of HySure due toits empirically observed robust and high performance. The
convex minimization problem is solved by the split augmented
Lagrangian shrinkage algorithm (SALSA) [65], which is an
instance of the alternating direction method of multipliers(ADMM) [66].

H. MAP-SMM
The MAP-SMM algorithm [36] adopted an SMM to estimate the underlying spectral scene statistics or, more specifically, the conditional mean vector and covariance matrix ofthe high-resolution HS image with respect to the MS image.
The average spectrum, covariance matrix, and abundance mapof each endmember are derived from the low-resolution HS
image. A MAP objective function is formulated to optimize thehigh-resolution HS data relative to the input images based on
the SMM statistics. The MAP-SMM algorithm is performedin the principal component subspace of the low-resolution HS
image.

I. FUSEFUSE utilizes a Sylvester equation to solve the maximization problem of the likelihoods obtained from the forward observation models [59]. A closed-form solution for
the Sylvester equation improved computational performance.
FUSE can be generalized to Bayesian estimators by integrating
prior information into the fusion problem using the ADMM
and the block coordinate descent method. The relative SRF and
PSF are required as a priori knowledge and the optimization
procedure is conducted in a principal component subspace.
In this paper, naive Gaussian [59] and sparsity promoted
Gaussian [38] priors are used as prior information and thecorresponding algorithms are referred to as FUSE and FUSES, respectively.
IV. DATA SETS AND E VALUATION M ETHODOLOGYThis section first provides brief descriptions of the eight data
sets used in the experiments. Most of the data sets are publicly
available. To validate the generalizability and versatility of
the HS-MS fusion algorithms, we ensure the diversity of theobserved scenes (e.g., vegetation, mineral, and urban) andfusion scenarios. Next, the evaluation methodology is detailed,including simulation procedures and quality measures.

A. Data sets
Table II summarizes the main specifications of all data sets
used in this work, which are described below in more detail.
Moreover, RGB color composites of all data sets are shownin Fig. 4.

61) AVIRIS Indian Pines: This HS image is acquired by
the airborne visible/infrared imaging spectrometer (AVIRIS)
sensor over the Indian Pines test site in northwestern Indiana,
USA, in 1992 [67]. The AVIRIS sensor acquires 224 spectral
bands in the wavelength range from 0.4 to 2.5 µm with afull width at half maximum (FWHM) of 10 nm. The image
consists of 512⇥614 pixels at a GSD of 20 m. We selected a360⇥360 pixel size image (see Fig. 4(a)) and used 192 bands
after removing bands of strong water vapor absorption andlow SNRs.
2) AVIRIS Cuprite: This data set was acquired by the
AVIRIS sensor over the Cuprite mining district in Nevada,USA, in 1995.1 The entire data set comprises five reflectanceimages, and we used one of them saved in the file named
f970619t01p02 r02 sc03.a.rfl. The full image consists of
512⇥614 pixels at a GSD of 20 m. We selected a 420⇥360
pixel size image (see Fig. 4(b)) and used 185 bands after
removing noisy bands.
3) AVIRIS Moffett Field: This data set was acquired by the
AVIRIS sensor over the Moffett field in California, USA, in
1997.1 This calibrated radiance image consists of 1923⇥753
pixels at a GSD of 17 m. We selected a 360⇥360 pixel size
image (see Fig. 4(c)) and used 182 bands after removing water
absorption bands.
4) HYDICE Washington DC Mall: This image was takenby the hyperspectral digital imagery collection experiment(HYDICE) sensor over the Mall in Washington, DC, USA,in 1995. The HYDICE sensor acquired 210 bands between
0.4 to 2.5 µm, and the image consists of 1280⇥307 pixels at
a GSD of 2.5 m. We selected a 420⇥300 pixel size image (seeFig. 4(d)) and used 191 bands after removing water absorption
bands in the 0.9–1.4 µm region.
5) Hyperspec-VNIR Chikusei: The airborne HS data set
was taken by Headwall’s Hyperspec-VNIR-C imaging sensor
over Chikusei, Ibaraki, Japan, on July 29, 2014. The data
set comprises 128 bands in the spectral range from 0.363
to 1.018 µm. The scene consists of 2517 ⇥ 2335 pixels
with a GSD of 2.5 m, mainly including agricultural andurban areas. We selected a 540⇥420 pixel size image for theexperiment (see Fig. 4(e)). The data set was made available
to the scientific community recently, including more detailed
descriptions regarding the data acquisition and processing
details, in [68].
6) ROSIS-3 University of Pavia: This data was acquired by
the reflective optics spectrographic imaging system (ROSIS-3)
optical airborne sensor over the University of Pavia, Italy, in
2003. It consists of 610 ⇥ 340 pixels with a GSD of 1.3 m. The
12 noisy bands have been removed, so that a total of 103 bands
covering the spectral range from 0.430 to 0.838 µm are usedin the experiment. The image is available as reflectance data,
and a 560⇥320 pixel size image (see Fig. 4(f)) was selected.
7) CASI University of Houston: This data set was used in
the 2013 IEEE GRSS data fusion contest [69]. The image was
acquired by an ITRES CASI-1500 sensor over the Universityof Houston campus and its neighboring urban areas, with 144
bands covering the wavelength range from 0.364 to 1.046 µm.
1 [Online]. Available here: http://aviris.jpl.nasa.gov/data/free data.html

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

7TABLE IIDATA SET SPECIFICATIONS
Index
Site
Original sensor
Year
Data type
Spectral range (µm)
Bands
Used Bands
GSD
Spatial dim.
Multispectral sensor
GSD ratio
SNR (dB)

1/a
Indian Pines
AVIRIS
1992
Radiance
0.4–2.5224
192
20360⇥360
WV-3
435(a) AVIRIS Indian Pines(e) Hyperspec Chikusei

2/b
CupriteAVIRIS
1995
Reflectance
0.4–2.5224
185
20
420⇥360
WV-3
5
35

3/c
Moffett Field
AVIRIS
1997
Radiance
0.4–2.5224
182
17360⇥360
QB
4
35

4/d
Washington DC
HYDICE1995
DN
0.4–2.5
210
191
2.5
420⇥300
QB435(b) AVIRIS Cuprite(f) ROSIS-3 University of Pavia

5/eChikuseiHyperspec
2014
Reflectance
0.36–1.02
128
128
2.5
540⇥420
WV-2
6
35

6/f
Univ. Pavia
ROSIS-3
2003
Reflectance
0.43–0.84
115
103
1.3
560⇥320QB835(c) AVIRIS Moffett Field

7/g
Univ. Houston
CASI
2012
Radiance
0.36–1.05
144
144
2.5
320⇥540
Sentinel-2
5
35

8/h
Rodalquilar
HyMap
2003
Reflectance
0.4–2.5242 (126)
167
10 (4)
261⇥867
Sentinel-23–(d) HYDICE Washington DC Mall(g) CASI University of Houston(h) HyMap Rodalquilar

Fig. 4. Color composite images of (a) AVIRIS Indian Pines, (b) AVIRIS Cuprite, (c) AVIRIS Moffett Field, (d) HYDICE Washington DC Mall, (e) Hyperspec
Chikusei, (f) ROSIS-3 University of Pavia, (g) IEEE GRSS DFC 2013 University of Houston, and (h) HyMap Rodalquilar data sets.

8

Spectral Response

Spectral ResponseIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

1
0.5
0

0.5

1

1.5
Wavelength (Micrometer)

2

1
0.5
0

0.5

1

Spectral Response

Spectral Response

1
0.5
0.5

1

1.5
Wavelength (Micrometer)

2

1
0.5
0

0.5

1

0.5
0.5

0.6
0.7
0.8
Wavelength (Micrometer)

0.9

1Spectral Response

Spectral Response

1

0.4

0.5
0

0.5

0.6
0.7
0.8
Wavelength (Micrometer)

0.91(g) CASI University of Houston

Spectral Response

Spectral Response

0.5
0.5

0.6
0.7
Wavelength (Micrometer)

0.8

0.9

(f) ROSIS-3 University of Pavia

1

0.4

2

1

(e) Hyperspec Chikusei

0

1.5
Wavelength (Micrometer)(d) HYDICE Washington DC Mall(c) AVIRIS Moffett Field

02(b) AVIRIS Cuprite(a) AVIRIS Indian Pines

0

1.5
Wavelength (Micrometer)

1
0.5
0

0.5

1

1.5
Wavelength (Micrometer)2(h) HyMap Rodalquilar

Fig. 5. Center wavelengths of HS bands (blue bars) and SRFs of MS imagers (red) for the (a) AVIRIS Indian Pines, (b) AVIRIS Cuprite, (c) AVIRIS
Moffett Field, (d) HYDICE Washington DC Mall, (e) Hyperspec Chikusei, (f) ROSIS-3 University of Pavia, (g) CASI University of Houston, and (h) HyMap
Rodalquilar data sets.

It consists of 349 ⇥ 1905 pixels with a GSD of 2.5 m. We
selected a 320⇥540 pixel size image (see Fig. 4(g)).
8) HyMap Rodalquilar: The HyMap image was acquired
over Rodalquilar, Spain, in June 2003 [70]. The HyMap sensor
collected 126 bands in the 0.4–2.5 µm wavelength range. The
scene covers a gold mining area in the Sierra del Cabo deGata (Cabo de Gata National Park) with a GSD of 4 m. To
use this data set for a realistic simulation scenario of EnMAP
and Sentinel-2 data fusion, the HyMap image was spatiallydowngraded to a 10-m GSD and spectrally upsampled with
242 bands using linear interpolation and resampling using
EnMAP SRFs. The full simulation procedure used for thistest data set differs from the other seven data sets by meansof the end-to-end-simulation procedure [71], [72] described in
Section IV-B2. We selected a 261⇥867 pixel size image (seeFig. 4(h)) and used 167 bands after removing water absorption
bands.
B. Simulation procedures
An accurate quality assessment of fusion data can be
performed only within a simulation study [73]. The entire
quality assessment procedure mainly comprises three steps: 1)
simulate input HS-MS images from a reference high-resolution
HS image; 2) generate a high-resolution HS image by fusing
the two input images; 3) compare the fused image and thereference image.
The eight HS images described in Section III-A were usedas the reference images after denoising. In the literature,

the original high-resolution HS images were often used as
reference to conduct quality assessment; however, many ofthem include noise, which is not suitable as reference. To
improve the reliability of quantitative evaluation, a denoising
method in [74] is applied to the original images to increase
SNRs of reference images.
For the simulation of the input images, we adopt two
simulation strategies: 1) a standard simulation that considers
spectral simulation, spatial simulation, and noise simulation;2) an end-to-end simulation that takes into account the entire
image acquisition and processing chain starting from raw data.
The first strategy is used for the data sets #1–7 and the second
strategy is used for the data set #8.
1) Standard simulation: The flow diagram of the evaluation
methodology with the standard simulation is shown in Fig. 6.
In the standard simulation, spectral simulation is performed togenerate the MS image by degrading the reference image in the
spectral domain using MS SRFs as filters. For the diversity of
MS sensors, SRFs of four MS imagers were used for spectral
simulation, namely, WorldView-2 (WV-2) for the data set #5,
WorldView-3 (WV-3) for the data sets #1 and 2, QuickBird(QB) for the data sets #3, 4, and 6, and Sentinel-2 VNIR bands
at a 10 m GSD for the data set #7 (see Table II). Fig. 5 shows
overlaps of SRFs between the HS-MS imagers for all data sets.
For the data sets #1, 2, 5, and 6, the MS SRFs evenly covermost of the spectral range of the HS imager. In contrast, there
is no high-resolution MS image in the SWIR range for thedata sets #3, 4, and 8. The latter case is more challenging andIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

even though the 20 m GSD SWIR images could be potentiallyused additionally for enhancing the EnMAP image which isof 30 m GSD.

High-resolution HS image
Denoising
Blurring anddownsampling

Spectral response

Add noise

Add noise

Observed HS image

Observed MS image

HS-MS fusion
Estimated high-resolution
HS image
Denoising

9

Quality measures

Fig. 6. Flow diagram of the evaluation methodology, including the standard
simulation procedure (used e.g. in [56]).

was included in the experiment to investigate the impact of
the SRF overlap between HS and MS imagers on the qualityof fused data. Spatial simulation was performed to generate
the low-resolution HS image using an isotropic Gaussian PSF
with a FWHM of the Gaussian function equal to the resolution
ratio between the GSDs of both input images. Five different
GSD ratios, namely 3, 4, 5, 6, and 8, were included for spatial
simulation in the standard simulation (see Table II) to simulate
multiple realistic combinations of spaceborne HS-MS sensors,
shown in Fig. 2. After spectral and spatial simulations, banddependent Gaussian noise was added to the simulated HS-MS
images. For realistic noise conditions, an SNR of 35 dB was
simulated in all bands.
2) End-to-end simulation: On the basis of the data set #8,
EnMAP and Sentinel-2 L2a (orthorectified surface reflectancedata) products were simulated using the sensor end-to-end
simulation tools EeteS [71] and S2eteS [72]. These tools
comprise forward and backward simulators, which simulate
the data acquisition procedure and the calibration and preprocessing chain, respectively, from spatially and spectrally
oversampled data to the final EnMAP and Sentinel-2 products [75]. The forward simulator consists of four independent
atmospheric, spatial, spectral, and radiometric modules. The
spatial and spectral modules include resampling an image inthe spatial and spectral domains using the sensor-specific PSFs
and SRFs, respectively. The radiometric module transformed
the at-sensor radiance to DN by simulating instrumental noiseand calibration coefficients. The backward simulator consistsof calibration modules such as non-linearity, dark current, andabsolute radiometric calibration and pre-processing modules
such as radiometric calibration and atmospheric correction.
Compared to the standard simulation, the end-to-end simulation can generate more realistic data sets that include
errors such as sensor-specific noise and residual errors of
atmospheric correction. Sentinel-2 VNIR images with a GSD
of 10 m (bands 2, 3, 4, and 8) were used as the MS data only,

C. Quality measures
We use the following four complementary and widely used
quality measures for the quantitative fusion assessment: 1)
peak signal-to-noise ratio (PSNR); 2) spectral angle mapper(SAM); 3) erreur relative globale adimensionnelle de synthèse(ERGAS); 4) Q2n . This section briefly describes these measures and their characteristics.
Let X 2 RB⇥P denote the reference HS image with B
spectral bands and P pixels. X = [x1 , ..., xB ]T = [x1 , ..., xP ],where xi 2 RP ⇥1 is the ith band (i = 1, ..., B) and xj 2
RB⇥1 is the spectral signature of the jth pixel (j = 1, ..., P ).
X̂ denotes the estimated HS image.
1) PSNR: PSNR is used to evaluate the spatial reconstruction quality of each band. PSNR is the ratio between the
maximum power of a signal and the power of residual errors.
The PSNR of the ith band is defined as✓◆
max(xi )2,(1)
PSNR(xi , x̂i ) = 10 · log10
kxi x̂i k22 /P

where max(xi ) is the maximum pixel value in the ith reference band image. A larger PSNR value indicates a higher
quality of spatial reconstruction (for identical data, the PSNR
is infinite). Since the PSNR is a normalized value definedat each band, it can fairly evaluate reconstruction errors of
all bands without being affected by the absolute value of thedata. We use the average PSNR with respect to bands for thequality index of the entire fused image. We also investigate
the PSNR plot over all bands to analyze the band dependent
characteristics of the fused image.
2) SAM: The SAM index [76] is commonly used to quantify the spectral information preservation at each pixel. Moreprecisely, SAM determines the spectral similarity (or distance)by calculating the angle between two vectors of the estimatedand reference spectra. The SAM index at the jth pixel isdefined as!
xTj x̂j
SAM(xj , x̂j ) = arccos.
(2)
kxj k2 kx̂j k2
SAM values near zero indicate high spectral quality. We use
the average SAM value with respect to pixels for the quality
index of the entire data set. The spatial distribution of SAM
is used below to visualize location- or material-dependent the
spectral quality of the fusion results.
3) ERGAS: ERGAS provides a global statistical measureof the quality of the fused data [77] with the best value at 0,
which is defined as
v
u
B
u1 X
kxi x̂i k22
ERGAS(X, X̂) = 100dt,(3)
B i=1 1 1T xi 2
PP

where d is the GSD ratio between the higher- and lower-spatial
resolution input images, and 1P = [1, ..., 1] 2 RP ⇥1 . ERGAS
calculates the band-wise normalized RMSE and multiplies itwith the ratio of GSD to account for the difficulty in the fusion
problem.

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

104) Q2n : Q2n is a generalization of the universal image
quality index (UIQI) [78] and an extension of the Q4 index [79] to hyperspectral images based on hypercomplex numbers [80]. The UIQI or the “Q” index was proposed by Wang
and Bovik [78] to measure any distortion in monochromatic
images as the product of three factors: loss of correlation,
luminance distortion, and contrast distortion. The UIQI between one reference image band (x) and its corresponding
target image band (y) is defined as
Q(x, y) =2x+(

4 xy x̄ȳ,222y )(x̄ + ȳ )

(4)

where
P
P
1 X
1 X
xj , ȳ =
yj .
P j=1
P j=1
v
u
P
u1 X
x̄)2 , y = t(yj ȳ)2 ,
P j=1

x̄ =

v
u
P
u1 X
t
=
(xj
x
P j=1

P

xy =

1 X
(xj
P j=1

x̄)(yjȳ).

UIQI can be rewritten as
Q(x, y) =

xy

2x̄ȳ

x y

x̄2 + ȳ 2

2 x y
2
2
x+ y

(5)

The three components correspond to correlation, luminancedistortion, and contrast distortion, respectively. UIQI has been
designed for monochromatic images. To overcome this limitation and additionally take into account spectral distortion,
the Q4 index has been developed for four-band images bymodeling each pixel vector as a quaternion [79]. Q2n further
extends the Q4 index based on modeling each pixel spectrum(xj ) as a hypercomplex number, namely a 2n -ons that isrepresented as
xj = xj,0 + xj,1 i1 + xj,2 i2 + ... + xj,2n 1 i2n 1 .

(6)

Q2n can be computed using the hypercomplex correlation
coefficient, which jointly quantifies spectral and spatial distortions [80].
V. R ESULTS AND DISCUSSION
A. Experimental setting
To conduct a fair comparison, the fusion performance ofeach method was maximized by careful parameter tuning andoptimal algorithm setting for each data set. The MATLAB
source codes of all the algorithms are publicly available.2
Parameter tuning and algorithm setting were performed asfollows.
2 The source codes of GSA, SFIM, GLP, and HySure areavailableat
http://www.openremotesensing.net/index.php/codes/11pan-sharpening.
The
CNMF
code
is
availableat
http://naotoyokoya.com/Download.html. The ECCV’14 code is available at
http://www.csse.uwa.edu.au/⇠ajmal/code/HSISuperRes.zip. The ICCV’15
code is available at https://www1.ethz.ch/igp/photogrammetry/people/lanaras.
The MAP-SMM code is available in [36]. The FUSE code is available at
http://wei.perso.enseeiht.fr/publications.html

The key process of hypersharpening, as used for the
methods SFIM-HS and GLP-HS, is to synthesize a highresolution image for each HS band using a linear regression of MS bands. We consider two options for alinear regression, 1) least squares and 2) nonnegative leastsquares, and adopt an option that shows better results foreach data set. As a result, the least squares method wasused for the data sets #5, 6, and 7, and the nonnegative
least squares method was used for the data sets #1, 2,
3, 4, and 8. This suggests that the nonnegative leastsquares method is suitable for data sets with limited SRF
overlaps, while the least squares method performs betterfor data sets with good SRF overlaps (see Fig. 5). Notethat band selection that assigns the most correlated MSband to each HS band was also tested along with thesynthesis of a high-resolution band and compared withhypersharpening; the performance of hypershaprpening
was better for all tested data sets, which confirmed
the effectiveness of hypersharpening for pan-sharpening
based methods.
• We used estimated SRFs obtained by the nonnegative
least squares method [81] for all the methods that require
SRF as input (see Table I). Additional constraints, suchas smoothness of SRFs or consistency with pre-launch
SRFs, may be useful for estimating the relative SRF fromreal data sets [39], [82]; however, the nonnegative leastsquares method was adopted because it empirically led
to better fusion results.
• A Gaussian filter with an FWHM at GSD was used for
all the methods that require PSF except HySure. HySure
uses a PSF estimation method described in [39].
• The number of endmembers (k) is a key parameter for
unmixing-based methods. In this paper, k was set to 30
for CNMF, ICCV’15, and HySure as this number demonstrate stable and consistently competitive performance inall experiments and data sets.
• For ECCV’14, the number of atoms in the dictionary (k)was set to the number of HS bands (Bh ), the number ofatoms selected in each iteration of G-SOMP+ (L) was set
to 20, and a parameter for modeling error (⌘) in dictionary
learning was set to 10 3 after parameter tuning. Note that
dictionary learning doesn’t work well if we set ⌘ as a verysmall value (e.g., 10 8 as is in the original code).
• ICCV’15 originally used SISAL to initialize endmember
spectra. We also investigated its performance with the useof VCA and adopted the one that showed better resultsfor each data set. As a result, SISAL was used for thedata sets #2, 5, 6 ,7, and 8, while VCA was used for thedata sets #1, 3, and 4.
• There are several key parameters for MAP-SMM, such as
the number of subspace (k), the number of endmembers(ne ), and the number of mixture classes (nm ). In thispaper, we set k = 6, ke = 4, and km = 35, which areexperimentally confirmed as proper numbers. Note that
if the number of endmembers is large, the number ofsample spectra for a certain endmember decreases. This
results in a singular endmember covariance matrix, whichmakes the SMM process impossible.
•

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINETABLE IIIA LGORITHM - SPECIFIC MAIN PARAMETERS .
MethodGSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE / FUSE-S•

Parameter–––k = 30k = Bh , L = 20, ⌘ = 10 3
k = 30k = 30k = 6, ne = 4, nm = 35
6  k  15

The subspace dimension (k) is a critical parameter for
FUSE / FUSE-S. We select the optimal number for eachdata set maximizing the fusion performance. Dependingon the data set, parameter tuning has shown peak performance usually within a range of 6  k  15.

B. Visual analysis
First, we visually evaluate the quality of the fused images.
In Figs. 7–14, color composite images of the reference andinput HS images with three selected bands are shown in thefirst row and those of the fusion results are presented in thefirst column of the second to eleventh rows. Most of the fusion
methods considered in this paper provide good results and it ishard to visually discern the differences between the reference
and fused images. Therefore, the errors of the fused images arevisualized with three options, namely error (difference) imagesof color composites after contrast stretching, pixel-wise rootmean-square error (RMSE) images, and SAM images fromthe second to fourth columns, respectively, in the second toeleven rows of Figs. 7–14. The color-composite error images
visualize the spatial distribution of errors in three selectedbands, where gray pixels mean no fusion error and coloredpixels indicate local spectral distortion. It is important toreveal error characteristics of individual bands because they
have a major impact on many applications based on spectral
indices and band ratios that select very few specific bands.
The RMSE images visualize the magnitude of the error ateach pixel spectrum. The SAM images visualize the spatial
distribution of spectral angle errors. From the results, we makesix observations.
1) The methods operating on subspaces (GSA, CNMF,
ECCV’14, ICCV’15, MAP-SMM, FUSE, and FUSES) show material dependent errors resulting in visible
spatial structures in the color-composite error images.
This is because the accuracy of the fusion process canbe determined at each pixel by the reconstruction errorof each spectral signature via subspace representation.
2) The color-composite error images of GSA, SFIM-HS,
GLP-HS, and MAP-SMM include mainly random patterns as to be seen in Figs. 7 and 8. Since thosefour methods do not include implicit denoising in theiralgorithms, noise can be contained in the fused images
if the denoising post-processing fails to remove all thenoise. The denoising post-processing is important forthese four methods. Its impact is investigated below in
Section IV-D.

113) SFIM-HS, MAP-SMM, and FUSE show block error
patterns in Figs. 11, 12, and 13. This can be explained by the fact that those three methods directly
use information obtained from the low-resolution HS
image or adopt local processing in a shifting window.
SFIM-HS estimates a high-resolution image by modulating an nearest-neighbor-upscaled low-resolution image.
MAP-SMM adopts statistical information of spectral
signatures based on classification of the low-resolution
HS image in the fusion process. FUSE optimizes thehigh-resolution coefficient matrix on a block-by-blockbasis [59].
4) GSA, SFIM-HS, and GLP-HS show edge error patterns,
particularly in Figs. 9 and 10. This is because those threemethods sharpen the low-resolution image by adding
spatial details obtained either by calculating the difference between the high-resolution image and the upscaled
low-resolution synthetic image (in the case of GSA) or
by multiresolution analysis of the high-resolution image(in the cases of SFIM-HS and GLP-HS).
5) ECCV’14 shows comparatively poor results even after
extensive parameter tuning. This is particularly evident
in the tests with data sets #3, 4, and 8 (see Figs. 9, 10,
and 14), where the SRF overlaps between the two
imagers are limited. This lack in performance can betracked down to the estimation of high-resolution abundances which is only based on the MS image andsparsity prior information. The estimation result is likely
unreliable because this sparse regression problem isseverely ill-posed.
6) In SAM images, all methods show spatial patterns andthey are correlated to each other for some of the data sets
(e.g., the data sets #5–8). SAM values are generally highfor pixels that correspond to materials with low intensity(DN, radiance, or reflectance) values due to low SNRs.

C. Quantitative evaluation
We further investigate the quality of the entire resolutionenhanced HS data via quantitative measures. The quality of the
fused images was assessed by using the four quality measuresdescribed in Section IV-C. Tables IV–VI show the quality
measures of the fused data sets obtained by the ten methods
for the data sets #1–3, #4–6, and #7–8, respectively. The best
results are shown in bold type and the second best results
are indicated with underlines. From the tables, we make fiveobservations.
1) The hypersharpening methods generally demonstratecompetitive performance. In particular, when the MS
SRFs cover the spectral range of the HS imager in abalanced manner and the number of MS bands is small(e.g., the data sets #5, 6, and 7), the hypersharpening
methods outperform many of the other methods.
2) Several subspace-based methods (CNMF, ICCV’15,
HySure, and MAP-SMM) show good performance forthe data sets #1 and 2, outperforming the hypersharpening methods. This is because the estimation accuracy

Reference & HS

RMSE (×10-3 )8.6

0SAM (degree)

5

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

Fig. 7. AVIRIS Indian Pines data (data set #1): (1st row) Color composite
images of reference and input HS images are displayed for a 240⇥240 pixels
sub-area using 2.20, 0.80, and 0.46 µm for red, green, and blue, respectively.
(2nd-11th rows) Color composites of fused images (1st column), error images
relative to the reference data visualized by differences of color composites(2nd column), RMSE (3rd column), and SAM images (4th column).

0

GSA

5

HySureSAM (degree)

MAP-SMM0FUSE

RMSE (×10 1 )
9.8

GSA

0

12

FUSE-S

Reference & HSIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

Fig. 8. AVIRIS Cuprite data (data set #2): (1st row) Color composite imagesof reference and input HS images are displayed for a 240⇥240 pixels sub-areausing 2.20, 1.6, and 0.57 µm for red, green, and blue, respectively. (2nd-11throws) Color composites of fused images (1st column), error images relative tothe reference data visualized by differences of color composites (2nd column),
RMSE (3rd column), and SAM images (4th column).

Reference & HS

RMSE (×10 2 )
2.9

0SAM (degree)

5

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

Fig. 9. AVIRIS Moffett Field data (data set #3): (1st row) Color composite
images of reference and input HS images are displayed for a 240⇥240 pixels
sub-area using 1.24, 0.86, and 0.40 µm for red, green, and blue, respectively.
(2nd-11th rows) Color composites of fused images (1st column), error images
relative to the reference data visualized by differences of color composites(2nd column), RMSE (3rd column), and SAM images (4th column).

0

GSA

5

HySureSAM (degree)

MAP-SMM0FUSE

RMSE (×10 2 )
3.5

GSA

0

13

FUSE-S

Reference & HSIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

Fig. 10. HYDICE Washington DC Mall data (data set #4): (1st row) Color
composite images of reference and input HS images are displayed for a240⇥240 pixels sub-area using 2.20, 1.60, and 1.01 µm for red, green, andblue, respectively. (2nd-11th rows) Color composites of fused images (1stcolumn), error images relative to the reference data visualized by differencesof color composites (2nd column), RMSE (3rd column), and SAM images(4th column).

Reference & HS

RMSE (×10-2 )1.7

0SAM (degree)

5

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

Fig. 11. Hyperspec Chikusei data (data set #5): (1st row) Color composite
images of reference and input HS images are displayed for a 240⇥240 pixels
sub-area using 0.80, 0.67, and 0.55 µm for red, green, and blue, respectively.
(2nd-11th rows) Color composites of fused images (1st column), error images
relative to the reference data visualized by differences of color composites(2nd column), RMSE (3rd column), and SAM images (4th column).

0

GSA

5

HySureSAM (degree)

MAP-SMM0FUSE

RMSE (×10-3 )
7.9

GSA

0

14

FUSE-S

Reference & HSIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

Fig. 12. ROSIS-3 University of Pavia data (data set #6): (1st row) Color
composite images of reference and input HS images are displayed for a240⇥240 pixels sub-area using 0.67, 0.57, and 0.46 µm for red, green, andblue, respectively. (2nd-11th rows) Color composites of fused images (1stcolumn), error images relative to the reference data visualized by differencesof color composites (2nd column), RMSE (3rd column), and SAM images(4th column).

Reference & HS

RMSE (×10-2 )4.3

0SAM (degree)

5

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15

SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

Fig. 13. CASI University of Houston data (data set #7): (1st row) Color
composite images of reference and input HS images are displayed for a240⇥240 pixels sub-area using 0.65, 0.55, and 0.45 µm for red, green, andblue, respectively. (2nd-11th rows) Color composites of fused images (1stcolumn), error images relative to the reference data visualized by differencesof color composites (2nd column), RMSE (3rd column), and SAM images(4th column).

0

GSA

5

HySureSAM (degree)

MAP-SMM0FUSE

RMSE (×10 2 )
9.6

GSA

0

15

FUSE-S

Reference & HSIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

Fig. 14. HyMap Rodalquilar data (data set #8): (1st row) Color composite
images of reference and input HS images are displayed for a 240⇥240 pixels
sub-area using 2.20, 1.20, and 0.66 µm for red, green, and blue, respectively.
(2nd-11th rows) Color composites of fused images (1st column), error images
relative to the reference data visualized by differences of color composites(2nd column), RMSE (3rd column), and SAM images (4th column).

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

16

48

55

50

46
45

50

45

40

40

42

PSNR (dB)

PSNR (dB)

PSNR (dB)

44

40
38

35

30
36
35

25

34
30

32
0.5

1
1.5
Wavelength (Micrometer)

2

0.5

1
1.5
Wavelength (Micrometer)

202(b) AVIRIS Cuprite(a) AVIRIS Indian Pines
55

46

55

44
42

40

PSNR (dB)

PSNR (dB)

PSNR (dB)

45
45
40

35

40
38
36
34

35

32

30

30

25

25

0.5

1
1.5
Wavelength (Micrometer)

2

30
0.4

(d) HYDICE Washington DC Mall

0.5

0.6
0.7
0.8
Wavelength (Micrometer)

0.9(e) Hyperspec Chikusei

55

1

28

0.5

0.6
0.7
Wavelength (Micrometer)

0.8(f) ROSIS-3 University of Pavia

55
50

50

GSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

45
PSNR (dB)45
PSNR (dB)

2

48

50

40

40
35

35
30
30

25

1
1.5
Wavelength (Micrometer)(c) AVIRIS Moffett Field

60

50

0.5

25

0.4

0.5

0.6
0.7
0.8
Wavelength (Micrometer)

0.9(g) CASI University of Houston

1

20

0.5

1

1.5
Wavelength (Micrometer)2(h) HyMap Rodalquilar

Fig. 15. Comparison of PSNRs between the ten HS-MS fusion algorithms for the (a) AVIRIS Indian Pines, (b) AVIRIS Cuprite, (c) AVIRIS Moffett Field,(d) HYDICE Washington DC Mall, (e) Hyperspec Chikusei, (f) ROSIS-3 University of Pavia, (g) CASI University of Houston, and (h) HyMap Rodalquilar
data sets.

of high-resolution subspace coefficients or abundances
increases as the number of MS bands increases.
3) The performance of the unmixing-based methods (except ECCV’14) is good and stable as a whole. In
particular, the unmixing-based methods outperform theBayesian methods when the SRF overlap is low (e.g.,
the data sets #3, 4, and 8). This result suggests that it isbetter to process the data fusion in the original spectral
space rather than the projected space for the spectral
ranges without MS bands.
4) The unmixing-based methods (except ECCV’14) show
their advantage in the spectral quality measured by SAM
for all data sets.
5) The performance of ECCV’14 is limited when there is alarge spectral range covered by the HS sensor but not by
the MS bands (data sets #3, 4, and 8), which confirms
the fifth observation in the visual analysis.
Fig. 15 shows PSNR plots over the HS bands. PSNR plots
reveal reconstruction errors in each band separately. Manyof the considered methods show comparable results. As an
overall trend, the PSNR values are very good (> 40 dB)
in the spectral range covered by the MS bands, whereas
they dramatically decrease in the spectral ranges not coveredby the high-resolution sensor (e.g., the SWIR range in thedata sets #3, 4, and 8). GSA shows low PSNR values inspectral regions missing MS bands (see Fig. 15(e)(f)(g)). Thisis because those spectral bands are usually lowly correlatedwith the corresponding high-resolution images. For instance,

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

D. Robustness analysis
The robustness of all methods under comparison against
noise, SRF overlap, GSD ratios, and misregistration are systematically analyzed as follows.
1) Noise robustness: Fig. 16 shows the impact of denoising to the overall fusion quality using PSNR and SAM as
quality measures. It demonstrates that the denoising postprocessing consistently improves the performance of GSA,
SFIM-HS, GLP-HS, and MAP-SMM while neither improving nor worsening the fusion results produced by the othermethods. This observation can be explained by the fact that
CNMF, ECCV’14, ICCV’15, HySure, FUSE, and FUSE-S
all operate on a subspace. This meas that the dimension of
their fusion result is only as large as the dimension of thecorresponding underlying subspace. In case of unmixing based
methods, this dimension is given by the pre-defined number ofendmembers. Any denoising method reduces the dimension of
the supposedly high dimensional noisy data to some extend.
Now, subspace-based methods naturally imply denoising powers already. Therefore, depending on the denoising techniqueat hand, post-denoising does not affect those fusion results.
On the other hand, this observation affirms that denoising
post-processing is essential in the evaluation methodology tofairly compare resolution-enhancement performance of different approaches, whenever HS-MS fusion simulations take into
account additive noise for input images.
2) SRF overlap: To further investigate the impact of SRF
overlap on the quality of fused images, HS-MS fusion was
performed on the AVIRIS Cuprite data using the following
three scenarios of WV-3 MS bands: 1) VNIR-SWIR 16 bands
2) VNIR 8 bands; and 3) SWIR 8 bands.
Figs. 17(a) and 17(b) presents PSNR and SAM, respectively,
with the three aforementioned scenarios of MS bands. As canbe expected, in the second and third scenarios, the fusion
performances of all methods decrease compared to those in
the first scenario. In particular, the Bayesian methods and
ECCV’14 show significant degradation in both PSNR and
SAM, whereas the pan-sharpening-based methods, CNMF,
ICCV’15, and HySure present relatively stable results. Thisfinding suggests that HS-MS fusion should be performedon the original hyperspectral features rather than linearlytransformed features such as principal components, if the SRF
overlap is relatively little.
3) GSD ratio: We examine the robustness of the ten HSMS fusion algorithms against the GSD ratio using the Hyperspec Chikusei data set. Figs. 17(c) and 17(d) show PSNR and

GLP-HS
CNMF

GSA
SFIM-HSHySure
MAP-SMM

ECCV’14
ICCV’154FUSE
FUSE-S

0

3.5

-0.05
-0.1

SAM (degree)

3

PSNR (dB)

GSA is simply applied to HS-MS fusion by solving multiple
pan-sharpening problems after grouping HS bands with respect
to one MS band at a time based on correlation analysis. Thereis room for improvement by performing more detailed spectralgrouping (e.g., [28], [31]) and synthesizing high-resolution
images as hypersharpening. The method of grouping remains
an open issue. ECCV’14 shows good PSNR values in the
spectral ranges covered by MS bands. On the other hand, they
dramatically decrease in the other spectral ranges, which ispossibly caused by inaccurate estimation of high-resolution
abundances.

17

2.5
2
1.5
1

-0.15
-0.2
-0.25
-0.3
-0.35
-0.4

0.5
0
1

-0.45

2

3

4

5

6

7

8

-0.5
1

2

3

4

5

Data set number

Data set number(a) PSNR(b) SAM

6

7

8

Fig. 16. Impact of denoising on quality assessment of fused data using (a)PSNR and (b) SAM.

SAM for the ten fused images with three cases of the GSD
ratio, i.e., three, six, and ten. As the GSD ratio increases,
the fusion performances decrease in general. For both quality
measures, the hypersharpening methods, CNMF, and HySure
show the most robust results.
4) Misregistration: The robustness against misregistration
is analyzed using the Hyperspec Chikusei data set. We simulated two cases of global misregistration between the input HSMS images by shifting one of them three and six pixels in thehigher resolution in both horizontal and vertical directions. The
fusion results are compared to those without misregistration.
We assumed that the reference image is co-registered to the
MS image.
Figs. 17(e) and 17(f) show PSNR and SAM for all methods
with the perfect image registration and the two cases of misregistration. The fusion performances significantly decreaseas the degree of misregistration increases, which suggests theimportance of image registration as preprocessing in practice.
Noticeably, ECCV’14 and ICCV’15 show relatively robust
performances against misregistration. Those two algorithms
optimize the high-resolution abundances based on the MS
image at the end of the fusion procedure. Therefore, the fused
images are spatially more consistent with the MS image. On
the other hand, the other fused images are biased to the HS
image, leading to higher sensitivity to misregistration giventhat the reference image is co-registered to the MS image.
The reasons for that are two-fold: the other subspace-basedalgorithms optimize the fused images so that the spatiallydegraded versions of the fused data are consistent with the
input HS image; the pan-sharpening-based algorithms add
spatial details to the input HS image.
E. Impact of HS-MS fusion on classification
Finally, the quality of the fused images is indirectly validated via pixel-wise classification, which is one of the most
relevant topics in the analysis of HS remote sensing data.
Since the University of Pavia and University of Houston
data sets have been widely used for validating classification
performance in the HS image processing community owing
to the availability of ground truth information, we perform
classification on those two data sets. The classification performances obtained using the fused images are compared toIEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

45

18

50

ŶVNIR-SWIRŶ VNIRŶ SWIR50

GSD ratio: Ŷ 3 Ŷ 6 Ŷ 10

Misregistration (pixel): Ŷ 0 Ŷ 3 Ŷ 6

45

40
40

35

PSNR (dB)

PSNR (dB)

PSNR (dB)

45

40

35
30

30

1.5

1

0.5

(b) Impact of SRF overlap on SAM

SE

SE
-S

FU

M

FU

e
AP
-

SM

M

15

ur

V’

yS

C
IC

H

F

14
V’
C

EC

S

M
N

G

C

LP

-H

S

SA

-H

6
4

(d) Impact of GSD ratio on SAM

-S

SE

SE

FU

FU

e

M
M
-S

M
AP

H

yS

ur

5
V’
1

IC

C

14
V’

EC

C

M
F
N
C

S

S

-H
LP
G

SA
G

0

IM
-H

-S

SE

SE

FU

FU

e

M
M
-S

M
AP

H

yS

ur

5
V’
1

IC

C

14
V’

EC

C

M
F
N
C

S
G

LP

-H

S

SA
G

IM
-H

0

8

2

SF

-S
SE

SE
FU

FU

e
ur

M
M

M

AP

-S

yS

V’
15
C

IC

H

F

14
V’
C

EC

M
N
C

S

S

-H
LP
G

-H
M

G

SA

0.5

Misregistration (pixel): Ŷ 0 Ŷ 3 Ŷ 6

10

SAM (degree)

1

SF
I

G

SF

12

GSD ratio: Ŷ 3 Ŷ 6 Ŷ 10

2

1.5

IM

SE

SE
-S

FU

FU

e
AP
-

SM

M

15

ur

V’

yS

C
IC

H

F

14
V’
C

S

M
C

N

S

SA

-H
LP
G

EC

M

(e) Impact of misregistration on PSNR

SF

2.5

ŶVNIR-SWIRŶ VNIRŶ SWIR2020(c) Impact of GSD ratio on PSNRSAM (degree)

SAM (degree)

2.5

-H

G

SF

(a) Impact of SRF overlap on PSNR
3

IM

SE

SE
-S

M

FU

35

FU

e

M

ur

SM
AP
-

V’

yS

C
IC

H

V’

15

14

F

G

EC

C

C

S

M
N

S

-H
LP

-H

SF

IM

G

25

SA

25

(f) Impact of misregistration on SAM

Fig. 17. Robustness of all methods under comparison against SRF overlap (left: (a) and (b)), GSD ratio (middle: (c) and (d)), and misregistration (right:(e) and (f)) measured by PSNR (top: (a), (c), and (e)) and SAM (bottom: (b), (d), and (f)) for Hyperspec Chikusei data. Fusion results with three scenarios
indicated by blue, green, and red bars are compared for each sensitivity analysis.

those obtained using the reference and input HS-MS images.
Support vector machines (SVMs) [83] and rotation forests(RoFs) [84] were adopted as classifiers because of their good
performance for classifying HS data with a small number oftraining samples [85], [86]. The class names and numbers of
training and test samples are summarized in Table VII. The
classification performance is quantitatively validated using theoverall accuracy (OA), the average accuracy (AA), and the
kappa coefficient (). Since training samples were randomly
selected, we repeated the classification test ten times.
Table VIII shows the results of average OA, AA, and 
under different trials for the University of Pavia and Universityof Houston data sets. Fig. 18 presents the classification mapsof one trial for the two data sets obtained by RoF using thereference, HS, MS, and HySure-fused images together with thetest sample maps. From the results, we make four observations:1) The classification accuracies obtained by RoF using
most of the fused images (except ECCV’14) are higher
than those using the input HS-MS images, which demonstrate the benefit of HS-MS fusion in the presentedclassification tasks. This is also visually demonstrated inFig. 18. The classification maps of HySure show muchbetter spatial details compared with those of the input
HS images. Although the classification maps of the input
MS images are spatially detailed, the classification mapsof HySure demonstrate better material discrimination

capabilities than those of the MS images as shown in,for example, meadows and self-blocking bricks in the
University of Pavia data set.
2) The average OA, AA, and  are largely consistent
with the fusion quality assessment results reported inTables V and VI. For example, HySure led to very goodclassification results whereas ECCV’14 resulted in poor
classification accuracies for both data sets.
3) Noticeably, for some methods the relative classification performance does not confirm the ranking in thestatistical quantitative assessment tests above. GSA,
for instance, showed a high potential for classification
purposes even though it was not competitive in thepreviously conducted quantitative assessment tests.
4) Some of the fused images led to higher classification
accuracies than those of the reference images. This ismainly due to the denoising effect of HS-MS fusion
or denoising post-processing. Higher SNRs of the fused
images can mitigate salt-and-pepper misclassified pixels.
It should be noted that classification-based validation of
HS-MS fusion results does not reflect the absolute quality of
fused data; firstly, because ground-truth information is only
available for a comparatively small spatial subset of the data,
and, secondly, because of the many factors and steps in thedata preprocessing chain that may influence the classification
capabilities relative to the ground-truth data. Characteristics

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

of applications and data analysis methods highly influence
the validation results. It is possible, for instance, to design
a fusion method specifically for one classification, application
or data analysis scenario for which it incorporates as muchinformation in its underlying mathematical fusion model asavailable. Dispite its presumably outstanding performancein that scenario, it would certainly fail in a more diverseand comprehensive testing setup like the one presented inthis paper. If one is interested in finding a fusion method
whose high-resolution data output is particularly suitable for
SVM or RoF-based classification, the performance comparisonpresented in this Section V-E provides a good starting point
for a more-indepth study tailored to the scenario of interest.
VI. C ONCLUSIONS AND F UTURE L INES
In this paper, we presented a comparative study of HSMS fusion methods for the resolution enhancement of HS
imagery. Ten state-of-the-art HS-MS fusion methods based onfour different approaches (CS, MRA, unmixing, and Bayesian)
were analyzed, assessed and compared using visual, quantitative, and classification-based means of evaluation. Publicly
available program codes were used while performing carefulparameter tuning at optimal individual algorithm settings. To
evaluate the generalizability and versatility of HS-MS-fusion
performance, eight simulated data sets were used in the experiments based on various representative fusion scenarios. The
overall fusion characteristics of all methods under comparison
regarding performance and robustness are summarized in Table IX. Those results are primarily derived from a quantitative
assessment of the fusion results under various fusion scenarios,
including variations in• imaging sensor types and sensor combinations,• spatial parameters such as the resolution ratio between thetwo input images and the fusion performance robustness
against misregistration in the data, and• spectral parameters such as the spectral coverage ofthe HS sensor (i.e., VNIR or VNIR-SWIR) and therelative spectral overlap with the MS sensor (i.e., different
numbers and spectral responsivities of MS bands).
It can be noted that, in the majority of fusion scenarios, theranking of the ten analyzed methods, which results from theindividual performance evaluations, does not change significantly. In particular, the visual and quantitative statistical comparisons are mostly consistent. Remarkably, the classificationbased quality assessment revealed that some of the fusion
methods, which showed good stable and robust overall performances throughout all tests based on statistical metrics,
did not achieve competitive scores in the classification-based
comparisons. The method which showed the most consistentand high performance in all tests including visual, statisticaland classification-based assessment is the Hyperspectral Superresolution (HySure) method by Simões et al. [39]. In theoverall comparison, HySure is closely followed by CNMF and,further, by the hypersharpaning adapted pan-sharpening-based
methods GLP-HS and SFIM-HS.
More specific findings regarding the individual methods
under comparison can be summarized as follows.

19

Algorithms that were originally designed for pansharpening and adopted to the HS-MS fusion problem
by hypersharpening are well competitive with methods
that are specifically developed for HS-MS data fusion. In
particular, the methods SFIM-HS and GLP-HS showedstable results above average in the majority of fusion
problems, especially when the spectral range of the HS
sensor is widely covered by MS bands.
• Unmixing-based methods (CNMF and HySure) demonstrated good and stable overall fusion performancethroughout all tests. Particularly, outstanding results are
obtained when the SRF overlap is limited, e.g., if a
VNIR-SWIR HS sensor is combined with a purely VNIRMS sensor. For both methods, caution is advised in thehigh-resolution abundances estimation step; outstanding
results could be achieved due to the minimization of theunmixing reconstruction errors with respect to both HSMS images rather than only the MS image (as donee.g. by ECCV’14). This difference made CNMF andHySure preserve the consistency between the HS andfused images better than ECCV’14. A clear advantage
of HySure over CNMF was observed in the classification
tests, in which HySure maintained its previously shown
performance superiority while CNMF revealed a flaw
not visible in the tests based on established assessment
criteria.
• Bayesian methods based on linear subspace transformations (FUSE and FUSE-S) showed good quantitative andvisual performance especially if the SRF overlap between
the two sensors is not too limited. A degradation in
performance can be observed when the spectral range ofthe HS sensor is largely uncovered by MS bands. The
classification capabilities are similar to, yet a little worse
than, those attained by CNMF.
• Additive noise in the simulated input HS-MS images
has a significant impact on the fusion quality as somemethods perform denoising implicitly while others do
not. Therefore, denoising is necessary for the evaluation methodology to compare the resolution-enhancement
performance of different algorithms fairly. This is rarely
done in the literature, which is one of the reasons why
many comparisons are not doubtlessly representative.
• Most of the presented methods (except ECCV’14)
demonstrated the benefit of HS-MS fusion in classification applications. That is, the classification accuracyof the fusion results exceed the accuracy achieve by theindividual input HS-MS images.
•

It is worth noting that hypersharpening and unmixingbased methods presented similar performances in terms ofquantitative quality measures, while different characteristics
of reconstruction errors were observable in the visual analysis.
A possible future direction for further performance improvement lies in developing hybrid approaches that combine theadvantages of different classes of methods [87], such as MRAbased hypersharpening and unmixing-based approaches [88].
Comprehensive modeling and compensation of realistic noiseand registration errors are still widely ignored in the fusion

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

community [89]. One reason for the superior performanceof HySure is the algorithm’s implicit denoising capacity.
However, there is still much potential for robustness improvement when accounting for more error sources, such as partial
coverage of one input image by clouds or seasonal differences
between the acquisitions. In addition, an optimal design of HSMS sensors capable of acquiring high-resolution HS data canbe of interest for future optical Earth observation missions, assuch data opens new fields of applications on a global scale,
which have been explorable only locally by airborne sensors
so far. Further research with real data sets will be needed
to verify the practicability of HS-MS fusion methods for theupcoming satellite missions.
ACKNOWLEDGMENT
The authors would like to thank Prof. D. Landgrebe from
Purdue University, West Lafayette, IN, USA, for providing
the AVIRIS Indian Pines data set, Prof. P. Gamba fromthe University of Pavia, Italy, for providing the ROSIS-3
University of Pavia data set, the Hyperspectral Image Analysis
group and the NSF Funded Center for Airborne Laser Mapping(NCALM) at the University of Houston for providing the
CASI University of Houston data set. The authors would liketo express their appreciation to Dr. P. Ghamisi and Dr. J. Xia
for providing MATLAB codes for classification.

20

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE21

TABLE IVQ UALITY MEASURES FOR AVIRIS I NDIAN P INES , C UPRITE , M OFFETT F IELD DATA S ETS
MethodGSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-SAVIRIS Indian Pines (data set #1)
PSNR
SAM
ERGAS
Q2n
40.0997
0.96775
0.447810.95950
40.74150.84069
0.400430.9129741.2962
0.82635
0.37533
0.95236
41.5609
0.642220.36194
0.92271
39.7953
1.10090.452270.90626
41.1837
0.782140.41192
0.97398
42.4427
0.62333
0.328160.9320242.7386
0.709080.33592
0.9780339.9802
0.83195
0.44434
0.8665340.0461
0.828840.442390.86613AVIRIS Cuprite (data set #2)
PSNR
SAM
ERGAS
Q2n
39.2154
0.982650.37458
0.9825440.8674
0.79776
0.31375
0.9701740.82400.802500.31570
0.9783842.97960.584430.25800
0.98498
38.80211.084
0.400640.95788
42.1053
0.632760.28777
0.9752643.72440.54640.23543
0.97883
41.4749
0.695670.30736
0.9734241.0701
0.78035
0.320120.97706
41.38070.746710.30958
0.97852AVIRIS Moffett Field (data set #3)
PSNR
SAM
ERGAS
Q2n
32.8652.09425.019
0.87563
34.3216
2.10745.24040.8482835.3179
1.96934.93780.856236.762
1.85224.01290.85706
27.9978
4.795420.0605
0.65486
35.6101
1.95654.95680.8351336.20311.75824.74620.8572234.52362.393
5.759
0.7513434.62022.448
5.20540.75063
35.19672.37274.82550.76018

TABLE VQ UALITY MEASURES FOR HYDICE WASHINGTON DC M ALL , H YPER S PEC C HIKUSEI , AND ROSIS-3 U NIVERSITY OF PAVIA DATA S ETS
MethodGSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

HYDICE Washington DC Mall (data set #4)
PSNR
SAM
ERGAS
Q2n
36.60092.44353.56250.9649738.09621.8035
3.23430.97287
38.1395
1.7661
3.11410.97451
38.6051.70653.13210.97465
34.5833
2.4785
6.61
0.92956
37.4458
1.78783.689
0.9708138.1268
1.80973.47370.968435.5975
2.2082
3.78340.9606136.93542.25753.47370.96468
37.8734
2.06753.25420.96876

HyperSpec Chikusei (data set #5)
PSNR
SAM
ERGAS
Q2n
41.5463
1.56571.46170.88574
46.59811.31891.26380.919346.9134
1.21811.2806
0.9499746.4972
1.24971.5042
0.9547643.9058
1.48291.67390.94491
42.4743
1.33651.70220.91982
47.0792
1.1772
1.50850.9523943.2669
1.31431.44540.913345.4068
1.4716
1.6136
0.9239446.6515
1.4246
1.58730.95185

ROSIS-3 University of Pavia (data set #6)
PSNR
SAM
ERGAS
Q2n
38.15563.7202
1.11550.7440442.23932.61870.76399
0.897242.34312.609
0.75526
0.8944642.1934
2.57970.795140.87335
40.1176
3.28641.012
0.84024
39.7306
2.7444
0.94646
0.77691
42.19882.77860.80787
0.8900939.9661
2.87570.91774
0.83684
41.75462.85120.83303
0.8830242.64842.69440.77587
0.90354

TABLE VIQ UALITY MEASURES FOR CASI U NIVERSITY OF H OUSTON AND H Y M AP RODALQUILAR DATA S ETS
MethodGSA
SFIM
GLP-HSCNMF
ECCV’14
ICCV15
HySure
MAP-SMMFUSE
FUSE-SCASI University of Houston (data set #7)
PSNR
SAM
ERGAS
Q2n
45.0846
2.0723
1.16420.9857747.2461
1.49611.016
0.9868747.2508
1.47331.01690.9869747.10191.526
1.3246
0.97505
44.03132.39292.288
0.9650744.1182
1.46271.2901
0.98477
47.6637
1.4437
1.11970.9835345.3191.6696
1.16590.98408
46.3679
1.68941.23150.97896
46.7968
1.59711.159
0.98073

HyMap Rodalquilar (data set #8)
PSNR
SAM
ERGAS
Q2n
40.20132.65971.938
0.63895
38.47782.66922.08110.58584
38.48832.65062.07150.59433
41.03612.71092.05490.70558
35.5714.6443
3.99140.54574
41.0904
2.6723
1.92850.64059
41.24542.67871.90780.70869
37.1782.71152.259
0.51056
38.41232.71542.142
0.6095438.8257
2.69142.078
0.60131

TABLE VIIC LASS NAMES AND NUMBERS OF TRAINING AND TEST SAMPLES FOR THE U NIVERSITY OF PAVIA AND U NIVERSITY OF H OUSTON DATA SETS .
Data
No.
1
2
3
4
5
6
7
8
9
10
11

University of Pavia
Name
Train
Asphalt
10
Meadows
10
Gravel10Trees
10Painted metal sheets
10
Bare soil
10
Bitumen10
Self-blocking bricks
10
Shadows
10
–
–
–
–

Test
5577
9920
2066
2655
1335
5019
1320
3590
937
–
–

University of Houston
Name
Train
Test
Healthy grass
9384
Stressed grass
9
199
Trees
9
507
Soil
9
227
Water
9
143
Residential
9165
Commercial
9
308
Road
9
302
Highway9118Parking lot 1
9365
Parking lot 2
9215

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

22TABLE VIIIOVERALL CLASSIFICATION ACCURACY FOR THE ROSIS-3 U NIVERSITY OF PAVIA AND CASI U NIVERSITY OF H OUSTON DATA SETS .
Data set
Classifier
Accuracy
Reference
HS
MS
GSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

OA
69.51
70.03
64.66
75.35
71.91
74.34
69.32
63.70
72.05
74.02
71.95
71.10
69.42

SVM
AA
76.92
70.45
71.82
80.77
77.82
80.03
76.65
72.80
78.24
80.04
79.04
77.76
76.94

University of Pavia

63.59
64.22
57.92
70.63
66.49
69.39
63.62
57.19
66.72
69.10
66.68
65.54
63.64

OA
70.35
65.85
64.79
80.41
74.79
75.67
74.27
64.62
76.14
80.45
74.31
74.17
73.72

RoF
AA
76.94
67.15
72.39
84.44
80.18
81.22
78.95
71.69
80.85
84.94
80.15
79.44
79.04


64.77
59.27
58.30
76.61
70.02
71.10
69.35
58.17
71.59
76.62
69.43
69.27
68.77

OA
74.54
73.65
71.71
72.20
71.96
72.21
72.27
70.13
75.10
75.18
71.09
71.67
71.29

University of Houston
SVM
RoF
AA

OA
AA
72.95
71.51
78.47
77.06
72.55
70.56
67.56
66.11
70.59
68.40
71.70
70.12
70.81
68.90
77.07
76.18
70.35
68.65
73.29
71.67
70.38
68.89
74.33
72.79
71.15
69.01
74.57
73.10
69.05
66.64
69.64
67.58
73.60
72.15
74.92
73.17
73.32
72.22
79.48
77.91
70.12
67.71
74.14
72.88
70.66
68.36
73.47
72.14
69.94
67.90
73.16
71.73


75.86
63.84
68.33
74.34
70.10
71.26
71.54
66.01
71.93
77.03
71.05
70.32
69.96

Pavia UniversityŶ AsphaltŶ Meadows
Ŷ GravelŶ TreesŶ Painted metal sheetsŶ Bare soilŶ BitumenŶ Self-blocking bricks
Ŷ Shadows

Test samples

Reference (OA: 72.6%)

HS (OA: 62.4%)

MS (OA: 64.6%)

HySure (OA: 83.7%)

Houston University

Ŷ Healthy grassŶ Stressed grass
Ŷ Trees
Ŷ SoilŶ WaterŶ Residential
Ŷ Commercial
Ŷ RoadŶ HighwayŶ Parking lot 1
Ŷ Parking lot 2

Test samples

Reference (OA: 74.8%)

HS (OA: 67.0%)

MS (OA: 69.9%)

HySure (OA: 78.5%)

Fig. 18. Classification maps of reference, HS, MS, and HySure images obtained by rotation forests for (top) University of Pavia and (bottom) University of
Houston data sets.
TABLE IXP ROS AND CONS OF ALL METHODS UNDER COMPARISON . O NE , TWO , OR THREE • SIGNS MEAN LOW, MEDIUM AND HIGH .
MethodGSA
SFIM-HS
GLP-HSCNMF
ECCV’14
ICCV’15
HySure
MAP-SMMFUSE
FUSE-S

General
••
••
•••
•••
•
••
•••
••
••
••

Performance
SpatialSpectral
••
••
•••
•••
•••
•••
•••
•••
••
•
••
•••
•••
•••
••
••
••
••
•••
••

Noise
•
•
•
••
••
••
•••
•
••
••

SRF
•••
•••
•••
•••
••
•••
•••
••
••
••

Robustness
GSD
Misreg.
••
•
•••
•
•••
•
•••
•
••
••
••
••
•••
•
•••
•
••
•
••
•

Material
•
•••
•••
••
•
••
••
••
••
••

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINER EFERENCES[1] L. Guanter, H. Kaufmann, K. Segl, S. Förster, C. Rogaß, S. Chabrillat,
T. Küster, A. Hollstein, G. Rossner, C. Chlebek, and et al., “The EnMAP
spaceborne imaging spectroscopy mission for earth observation,” Remote
Sens., vol. 7, pp. 8830–8857, 2015.
[2] A. Iwasaki, N. Ohgi, J. Tanii, T. Kawashima, and H. Inada, “Hyperspectral imager suite (HISUI) - Japanese hyper-multi spectral radiometer,”in Proc. IEEE IGARSS, Jul. 2011, pp. 1025–1028.
[3] P. Stefano, P. Angelo, P. Simone, R. Filomena, S. Federico, S. Tiziana,
A. U. C. Vincenzo, N. Acito, D. Marco, and et al., “The PRISMA hyperspectral mission: Science activities and opportunities for agriculture
and land monitoring,” in Proc. IEEE IGARSS, Jul. 2013, pp. 4558–4561.
[4] R. Green, G. Asner, S. Ungar, and R. Knox, “NASA mission to measureglobal plant physiology and functional types,” in Proc. Aerospace
Conference, Mar. 2008, pp. 1–7.
[5] S. Michel, P. P. Gamet, and M. J. Lefevre-Fonollosa, “HYPXIMA
hyperspectral satellite defined for science, security and defense users,”
in Proc. IEEE WHISPERS, Jun. 2011, pp. 1–4.
[6] A. Eckardt, J. Horack, F. Lehmann, D. Krutz, J. Drescher, M. Whorton,and M. Soutullo, “DESIS (DLR Earth sensing imaging spectrometerfor the ISS-MUSES platform,” in Proc. IEEE IGARSS, Jul. 2015, pp.
1457–1459.
[7] Sentinel-2 user handbook, 2nd ed., European Space Agency, Jul. 2015.
[8] “The digitalglobe constellation,” Brochure, DigitalGlobe, Inc., Westminster, CO, USA, 2016.
[9] RapidEye imagery product specifications, 6th ed., Planet Labs Inc., Jan.
2016.
[10] L. Loncan, L. B. Almeida, J. B. Dias, X. Briottet, J. Chanussot,
N. Dobigeon, S. Fabre, W. Liao, G. A. Licciardi, M. Simões, J. Y.
Tourneret, M. A. Veganzones, G. Vivone, Q. Wei, and N. Yokoya,“Hyperspectral pansharpening: a review,” IEEE Geosci. Remote Sens.
Mag., vol. 3, no. 3, pp. 27–46, Sep. 2015.
[11] A. Mookambiga and V. Gomathi, “Comprehensive review on fusion
techniques for spatial information enhancement in hyperspectral imagery,” Multidimensional Systems and Signal Processing, pp. 1–17, Apr.
2016.
[12] G. Palubinskas, “Model-based view at multi-resolution image fusion
methods and quality assessment measures,” International Journal of
Image and Data Fusion, pp. 203–218, May. 2016.
[13] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M.
Bruce, “Comparison of pan sharpening algorithms: Outcome of the 2006
GRS-S data fusion contest,” IEEE Trans. Geosci. Remote Sens., vol. 45,
no. 10, pp. 3012–3021, Oct. 2007.
[14] Q. Du, N. Younan, R. King, and V. Shah, “On the performance
evaluation of pan-sharpening techniques,” IEEE Geosci. Remote Sens.
Lett., vol. 4, no. 4, pp. 518–522, Oct. 2007.
[15] C. Thomas, T. Ranchin, L. Wald, and J. Chanussot, “Synthesis ofmultispectral images to high spatial resolution: A critical review offusion methods based on remote sensing physics,” IEEE Trans. Geosci.
Remote Sens., vol. 46, no. 5, pp. 1301–1312, May 2008.
[16] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, “25 yearsof pansharpening: A critical review and new developments,” in Signal
Image Processing for Remote Sensing, 2nd ed., C. H. Chen, Ed. Boca
Raton, FL: CRC Press, 7 2011, ch. 28, pp. 533–548.
[17] G. Vivone, L. Alparone, J. Chanussot, M. D. Mura, A. Garzelli,
G. Licciardi, R. Restaino, and L. Wald, “A critical comparison among
pansharpening algorithms,” IEEE Trans. Geosci. Remote Sens., vol. 53,
no. 5, pp. 2565–2586, May 2015.
[18] X. X. Zhu, C. Grohnfeldt, and R. Bamler, “Exploiting joint sparsity
for pansharpening: The j-sparsefi algorithm,” IEEE Transactions on
Geoscience and Remote Sensing, vol. 54, no. 5, pp. 2664–2681, May
2016.
[19] W. Carper, T. M. Lillesand, and P. W. Kiefer, “The use of intensityhue-saturation transformations for merging spot panchromatic and multispectral image data,” Photogramm. Eng. Remote Sens., vol. 56, no. 4,
pp. 459–467, Apr. 1990.
[20] C. A. Laben and B. V. Brower, “Process for enhancing the spatial
resolution of multispectral imagery using pan-sharpening,” Patent US
6 011 875 A, 2000.
[21] B. Aiazzi, S. Baronti, and M. Selva, “Improving component substitution
pansharpening through multivariate regression of MS + Pan data,” IEEE
Trans. Geosci. Remote Sens., vol. 45, no. 10, pp. 3230–3239, Oct. 2007.
[22] J. G. Liu, “Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial details,” International
Journal of Remote Sensing, vol. 21, no. 18, pp. 3461–3472, Jan. 2000.

23[23] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, “MTFtailored multiscale fusion of high-resolution MS and Pan imagery,”Photogramm. Eng. Remote Sens., vol. 72, no. 5, pp. 591–596, May
2006.
[24] S. Li and B. Yang, “A new pan-sharpening method using a compressedsensing technique,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 2,
pp. 738–746, Feb. 2011.
[25] X. X. Zhu and R. Bamler, “A sparse image fusion algorithm withapplication to pan-sharpening,” IEEE Trans. Geosci. Remote Sens.,
vol. 51, no. 5, pp. 2827–2836, May 2013.
[26] R. Gomez, A. Jazaeri, and M. Kafatos, “Wavelet-based hyperspectral
and multi-spectral image fusion,” in Proc. SPIE, vol. 4383, 2001, pp.
36–42.
[27] Y. Zhang and M. He, “Multi-spectral and hyperspectral image fusion
using 3-D wavelet transform,” J. Electron. (China), vol. 24, no. 2, pp.
218–224, Mar. 2007.
[28] Z. Chen, H. Pu, B. Wang, and G.-M. Jiang, “Fusion of hyperspectral
and multispectral images: A novel framework based on generalizationof pan-sharpening methods,” IEEE Geosci. Remote Sens. Lett., vol. 11,
no. 8, pp. 1418–1422, Aug. 2014.
[29] C. Grohnfeldt, X. X. Zhu, and R. Bamler, “Jointly sparse fusion of
hyperspectral and multispectral imagery,” in Proc. IEEE IGARSS, Jul.
2013, pp. 1–4.
[30] ——, “The J-SparseFI-HM hyperspectral resolution enhancement
method Now fully automated,” in Proc. IEEE WHISPERS, Jun. 2014,
pp. 1–4.
[31] ——, “Splitting the hyperspectral-multispectral image fusion problem
into weighted pan-sharpening problems – the spectral grouping concept,”
in 2015 7th Workshop on Hyperspectral Image and Signal Processing:
Evolution in Remote Sensing (WHISPERS), June 2015, pp. 1–4.
[32] M. Selva, B. Aiazzi, F. Butera, L. Chiarantini, and S. Baronti, “Hypersharpening: A first approach on SIM-GA data,” IEEE J. Sel. Topics Appl.
Earth Observ. Remote Sens., vol. 8, no. 6, pp. 3008–3024, Jun. 2015.
[33] M. T. Eismann and R. C. Hardie, “Application of the stochastic mixing
model to hyperspectral resolution enhancement,” IEEE Trans. Geosci.
Remote Sens., vol. 42, no. 9, pp. 1924–1933, Sep. 2004.
[34] R. C. Hardie, M. T. Eismann, and G. L. Wilson, “MAP estimation for
hyperspectral image resolution enhancement using an auxiliary sensor,”IEEE Trans. Image Process., vol. 13, no. 9, pp. 1174–1184, Sep. 2004.
[35] M. T. Eismann and R. C. Hardie, “Hyperspectral resolution enhancement
using high-resolution multispectral imagery with arbitrary response
functions,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 3, pp. 455–
465, Mar. 2005.
[36] M. T. Eismann, “Resolution enhancement of hyperspectral imagery
using maximum a posteriori estimation with a stochastic mixing model,”Ph.D. dissertation, Univ. Daton, Dayton, OH, May 2004.
[37] Y. Zhang, S. D. Backer, and P. Scheunders, “Noise-resistant waveletbased bayesian fusion of multispectral and hyperspectral images,” IEEE
Trans. Geosci. Remote Sens., vol. 47, no. 11, pp. 3834–3843, Nov. 2009.
[38] Q. Wei, J. M. B. Dias, N. Dobigeon, and J.-Y. Tourneret, “Hyperspectral
and multispectral image fusion based on a sparse representation,” IEEE
Trans. Geosci. Remote Sens., vol. 53, no. 7, pp. 3658–3668, Jul. 2015.
[39] M. Simões, J. B. Dias, L. Almeida, and J. Chanussot, “A convex
formulation for hyperspectral image superresolution via subspace-based
regularization,” IEEE Trans. Geosci. Remote Sens., vol. 53, no. 6, pp.
3373–3388, Jun. 2015.
[40] H. N. Gross and J. R. Schott, “Application of spectral mixture analysis
and image fusion techniques for image sharpening,” Remote Sens.
Environ., vol. 63, no. 2, pp. 85–94, Feb. 1998.
[41] G. D. Robinson, H. N. Gross, and J. R. Schott, “Evaluation of two
applications of spectral mixing models to image fusion,” Remote Sens.
Environ., vol. 71, no. 3, pp. 272–281, Mar. 2000.
[42] B. Zhukov, D. Oertel, F. Lanzl, and G. Reinhackel, “Unmixing-based
multisensor multiresolution image fusion,” IEEE Trans. Geosci. Remote
Sens., vol. 37, no. 3, pp. 1212–1226, May 1999.
[43] O. Berné, A. Helens, P. Pilleri, and C. Joblin, “Non-negative matrix
factorization pansharpening of hyperspectral data: An application to midinfrared astronomy,” in Proc. IEEE WHISPERS, Jun. 2010, pp. 1–4.
[44] N. Yokoya, T. Yairi, and A. Iwasaki, “Hyperspectral, multispectral, andpanchromatic data fusion based on non-negative matrix factorization,”in Proc. IEEE WHISPERS, Jun. 2011, pp. 1–4.
[45] ——, “Coupled non-negative matrix factorization (CNMF) for hyperspectral and multispectral data fusion: application for pasture classification,” in Proc. IEEE IGARSS, Jul. 2011, pp. 1779–1782.
[46] R. Kawakami, J. Wright, Y. Tai, Y. Matsushita, M. Ben-Ezra, andK. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in Proc. IEEE CVPR, Jun. 2011, pp. 2329–2336.

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE[47] J. Bieniarz, D. Cerra, J. Avbelj, P. Reinartz, and R. Müller, “Hyperspectral image resolution enhancement based on specral unmixing andinformation fusion,” in Proc. ISPRS, Sep. 2011, pp. 33–37.
[48] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix
factorization unmixing for hyperspectral and multispectral data fusion,”
IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2, pp. 528–537, Feb.
2012.
[49] E. Wycoff, T.-H. Chan, K. Jia, W.-K. Ma, and Y. Ma, “A non-negative
sparse promoting algorithm for high resolution hyperspectral imaging,”
in Proc. ICASSP, May 2013, pp. 1409–1413.
[50] N. Akhtar, F. Shafait, and A. Mian, “Sparse spatio-spectral representation for hyperspectral image super-resolution,” in Proc. ECCV, 2014,
pp. 63–78.
[51] B. Huang, H. Song, H. Cui, J. Peng, and Z. Xu, “Spatial and spectral
image fusion using sparse matrix factorization,” IEEE Trans. Geosci.
Remote Sens., vol. 52, no. 3, pp. 1693–1704, Mar. 2014.
[52] M. A. Bendoumi, M. He, and S. Mei, “Hyperspectral image resolution
enhancement using high-resolution multispectral image based on spectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 10, pp.
6574–6583, Oct. 2014.
[53] J. Bieniarz, R. Mller, X. X. Zhu, and P. Reinartz, “Hyperspectral image
resolution enhancement based on joint sparsity spectral unmixing,” inProc. IEEE IGARSS, 2014, pp. 2645–2648.
[54] C. Lanaras, E. Baltsavias, and K. Schindler, “Hyperspectral superresolution by coupled spectral unmixing,” in Proc. IEEE ICCV, Dec.
2015, pp. 3586–3594.
[55] ——, “Advances in hyperspectral and multispectral image fusion andspectral unmixing,” in Proc. ISPRS, vol. XL-3/W3, 2015, pp. 451–458.
[56] M. Veganzones, M. Simões, G. Licciardi, N. Yokoya, J. Bioucas-Dias,
and J. Chanussot, “Hyperspectral super-resolution of locally low rank
images from complementary multisource data,” IEEE Trans. Image
Process., vol. 25, no. 1, pp. 274–288, Jan. 2016.
[57] Q. Wei, N. Dobigeon, and J.-Y. Tourneret, “Bayesian fusion of multiband
images,” IEEE J. Sel. Topics Signal Process., vol. 9, no. 6, pp. 1117–
1127, Sep. 2015.
[58] N. Akhtar, F. Shafait, and A. Mian, “Bayesian sparse representation for
hyperspectral image super resolution,” in Proc. IEEE CVPR, 2015, pp.
3631–3640.
[59] Q. Wei, N. Dobigeon, and J.-Y. Tourneret, “Fast fusion of multi-band
images based on solving a Sylvester equation,” IEEE Trans. Image
Process., vol. 24, no. 11, pp. 4109–4121, Nov. 2015.
[60] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative
matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791, Oct.
1999.
[61] J. M. P. Nascimento and J. M. B. Dias, “Vertex component analysis: A
fast algorithm to unmix hyperspectral data,” IEEE Trans. Geosci. Remote
Sens., vol. 43, no. 4, pp. 898–910, Apr. 2005.
[62] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online dictionary learningfor sparse coding,” in Proc. ICML, 2009, pp. 1–8.
[63] J. Bioucas-Dias, “A variable splitting augmented lagrangian approach tolinear spectral unmixing,” in Proc. IEEE WHISPERS, 2009, pp. 1–4.
[64] J. Bioucas-Dias and M. Figueiredo, “Alternating direction algorithms forconstrained sparse regression: Application to hyperspectral unmixing,”
in Proc. IEEE WHISPERS, 2010, pp. 1–4.
[65] M. V. Afonso, J. Bioucas-Dias, and M. Figueiredo, “An augmented
lagrangian approach to the constrained optimization formulation ofimaging inverse problems,” IEEE Trans. Image Process., vol. 20, no. 3,
pp. 681–695, Mar. 2011.
[66] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed
optimization and statistical learning via the alternating direction methodof multipliers,” Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1–122,
Jan. 2011.
[67] M. F. Baumgardner, L. L. Biehl, and D. A. Landgrebe, “220 band
AVIRIS hyperspectral image data set: June 12, 1992 Indian Pine
test site 3,” Sep 2015. [Online]. Available: https://purr.purdue.edu/
publications/1947/1[68] N. Yokoya and A. Iwasaki, “Airborne hyperspectral data over Chikusei,”Space Application Laboratory, the University of Tokyo, Japan, Tech.
Rep. SAL-2016-05-27, May 2016.
[69] C. Debes, A. Merentitis, R. Heremans, J. Hahn, N. Frangiadakis, T. vanKasteren, W. Liao, R. Bellens, A. Piz̆urica, S. Gautama, W. Philips,
S. Prasad, Q. Du, and F. Pacifici, “Hyperspectral and LiDAR data fusion:
Outcome of the 2013 GRSS data fusion contest,” IEEE J. Sel. Topics
Appl. Earth Observ. Remote Sens., vol. 7, no. 6, pp. 2405–2418, Jun.
2014.

24

[70] E. Bedini, F. V. der Meer, and F. V. Ruitenbeek, “Use of HyMap
imaging spectrometer data to map mineralogy in the Rodalquilar caldera,
southeast Spain,” Int. J. Remote Sens., vol. 30, no. 2, pp. 327–348, 2009.
[71] K. Segl, L. Guanter, C. Rogass, T. Kuester, S. Roessner, H. Kaufmann,
B. Sang, V. Mogulsky, and S. Hofer, “EeteS The EnMAP end-to-end
simulation tool,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.,
vol. 5, no. 2, pp. 522–530, Apr. 2012.
[72] K. Segl, L. Guanter, F. Gascon, T. Kuester, C. Rogass, and C. Mielke,“S2eteS: An end-to-end modeling tool for the simulation of Sentinel-2
image products,” IEEE Trans. Geosci. Remote Sens., vol. 53, no. 10, pp.
5560–5571, Oct. 2015.
[73] L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images ofdifferent spatial resolutions: Assessing the quality of resulting images,”
Photogramm. Eng. Remote Sensing, vol. 63, no. 6, pp. 691–699, 1997.
[74] R. E. Roger and J. F. Arnold, “Reliably estimating the noise in AVIRIS
hyperspectral images,” Int. J. Remote Sen., vol. 17, no. 10, pp. 1951–
1962, 1996.
[75] N. Yokoya, J. C. W. Chan, and K. Segl, “Potential of resolution-enhanced
hyperspectral data for mineral mapping using simulated EnMAP and
Sentinel-2 images,” Remote Sensing, vol. 8, no. 3: 172, 2016.
[76] F. A. Kruse, A. B. Lefkoff, J. W. Boardman, K. B. Heidebrecht,
A. T. Shapiro, J. P. Barloon, and A. F. H. Goetz, “The spectral image
processing system (SIPS) - Interactive visualization and analysis ofimaging spectrometer data,” Remote Sensing of Environment, vol. 44,
pp. 145–163, 1993.
[77] L. Wald, “Quality of high resolution synthesised images: Is therea simple criterion?” in Proc. the Third Conference Fusion of Earth
Data: Merging Point Measurements, Raster Maps and Remotely Sensed
Images, Jan. 2000, pp. 99–103.
[78] Z. Wang and A. C. Bovik, “A universal image quality index,” IEEE
Signal Process. Lett., vol. 9, no. 3, pp. 81–84, Mar. 2002.
[79] L. Alparone, S. Baronti, A. Garzelli, and F. Nencini, “A global quality
measurement of pan-sharpened multispectral imagery,” IEEE Geosci.
Remote Sens. Lett., vol. 1, no. 4, pp. 313–317, Oct. 2004.
[80] A. Garzelli and F. Nencini, “Hypercomplex quality assessment ofmulti/hyperspectral images,” IEEE Geosci. Remote Sens. Lett., vol. 6,
no. 4, pp. 662–665, Oct. 2009.
[81] G. D. Finlayson, S. Hordley, and P. M. Hubel, “Recovering device
sensitivities with quadratic programming,” in Proc. IS and T/SID Sixth
Color Imaging Conference: Color Science, Systems, and Applications,
1998, pp. 90–95.
[82] N. Yokoya, N. Mayumi, and A. Iwasaki, “Cross-calibration for data
fusion of EO-1/Hyperion and Terra/ASTER,” IEEE J. Sel. Topics Appl.
Earth Observ. Remote Sens., vol. 6, no. 2, pp. 419–426, Apr. 2013.
[83] V. N. Vapnik, The Nature of Statistical Learning Theory. New York,NY, USA: Springer-Verlag New York, Inc., 1995.
[84] J. J. Rodriguez and L. I. Kuncheva, “Rotation forest: A new classifier
ensemble method.” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28,
no. 10, pp. 1619–1630, Oct. 2006.
[85] F. Melgani and L. Bruzzone, “Classification of hyperspectral remotesensing images with support vector machines,” IEEE Trans. Geosci.
Remote Sens., vol. 42, no. 8, pp. 1778–1790, Aug. 2004.
[86] J. Xia, P. Du, X. He, and J. Chanussot, “Hyperspectral remote sensing
image classification based on rotation forest,” IEEE Geosci. Remote
Sens. Lett., vol. 11, no. 1, pp. 239–243, Jan. 2014.
[87] C. Grohnfeldt and X. X. Zhu, “Towards a combined sparse representation and unmixing based hyperspectral resolution enhancement method,”in Proc. IEEE IGARSS, 2015, pp. 2872–2875.
[88] R. Guerra, S. López, and R. Sarmiento, “A computationally efficientalgorithm for fusing multispectral and hyperspectral images,” IEEE
Trans. Geosci. Remote Sens., vol. 54, no. 10, pp. 5712–5728, Jun. 2016.
[89] C. Zou and Y. Xia, “Poissonian hyperspectral image superresolution
using alternating direction optimization,” IEEE J. Sel. Topics Appl. Earth
Observ. Remote Sens., vol. 9, no. 9, pp. 4464–4479, Sep. 2016.

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

Naoto Yokoya (S’10–M’13) received the M.Sc. and Ph.D. degrees inaerospace engineering from the University of Tokyo, Tokyo, Japan, in 2010
and 2013, respectively.
From 2012 to 2013, he was a Research Fellow with Japan Society for
the Promotion of Science, Tokyo, Japan. Since 2013, he is an AssistantProfessor with the University of Tokyo. Since 2015, he is also an Alexandervon Humboldt Research Fellow with the German Aerospace Center (DLR),
Oberpfaffenhofen, and Technical University of Munich (TUM), Munich,Germany. His research interests include image analysis and data fusion inremote sensing.

Claas Grohnfeldt (S’12) received the Diploma degree in mathematics fromthe University of Bremen, Germany, in 2012. During his studies, he went
on exchange at the University of Warwick, England, where he studied
mathematics for one year.
Since March 2012, he has been working towards the Ph.D. degree with boththe DLR German Aerospace Center and the Technische Universitt Mnchen(TUM), Munich, Germany. He was awarded a Munich Aerospace doctoral
research scholarship in 2012. Mr. Grohnfeldt won the first price in the DLR
Science Slam competition and the IEEE IGARSS Best Paper Award in 2013
and 2016, respectively. Before his Ph.D. activities he was a research assistantat the chair Optimization and Optimal Control at the Universtiy of Bremen
and a full-time intern at Airbus Defense and Space, Ottobrunn, Germany.
He worked for the European Space Agency, Noordwijk, the Netherlands,
and conducted his diploma thesis with Daimler AG, Stuttgart, Germany.
His current research interests include signal processing in Earth observation,satellite data fusion, algorithm development and high performance computing.

Jocelyn Chanussot (M’04–SM’04–F’12) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology (Grenoble INP),
Grenoble, France, in 1995, and the Ph.D. degree from the Universit deSavoie, Annecy, France, in 1998. In 1999, he was with the Geography
Imagery Perception Laboratory for the Delegation Generale de l’Armement(DGA - French National Defense Department). Since 1999, he has been with
Grenoble INP, where he was an Assistant Professor from 1999 to 2005,
an Associate Professor from 2005 to 2007, and is currently a Professor ofsignal and image processing. He is conducting his research at the Grenoble
Images Speech Signals and Automatics Laboratory (GIPSA-Lab). His researchinterests include image analysis, multicomponent image processing, nonlinear
filtering, and data fusion in remote sensing. He has been a visiting scholar
at Stanford University (USA), KTH (Sweden) and NUS (Singapore). Since
2013, he is an Adjunct Professor of the University of Iceland. In 2015-2017,
he is a visiting professor at the University of California, Los Angeles (UCLA).
Dr. Chanussot is the founding President of IEEE Geoscience and Remote
Sensing French chapter (2007-2010) which received the 2010 IEEE GRS-S
Chapter Excellence Award. He was the co-recipient of the NORSIG 2006
Best Student Paper Award, the IEEE GRSS 2011 and 2015 Symposium Best
Paper Award, the IEEE GRSS 2012 Transactions Prize Paper Award and theIEEE GRSS 2013 Highest Impact Paper Award. He was a member of theIEEE Geoscience and Remote Sensing Society AdCom (2009-2010), in chargeof membership development. He was the General Chair of the first IEEE
GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution
in Remote sensing (WHISPERS). He was the Chair (2009-2011) and Cochairof the GRS Data Fusion Technical Committee (2005-2008). He was a memberof the Machine Learning for Signal Processing Technical Committee of theIEEE Signal Processing Society (2006-2008) and the Program Chair of theIEEE International Workshop on Machine Learning for Signal Processing,(2009). He was an Associate Editor for the IEEE Geoscience and Remote
Sensing Letters (2005-2007) and for Pattern Recognition (2006-2008). Since
2007, he is an Associate Editor for the IEEE Transactions on Geoscience andRemote Sensing. He was the Editor-in-Chief of the IEEE Journal of Selected
Topics in Applied Earth Observations and Remote Sensing (2011-2015). In
2013, he was a Guest Editor for the Proceedings of the IEEE and in 2014 a
Guest Editor for the IEEE Signal Processing Magazine. He is a Fellow of theIEEE and a member of the Institut Universitaire de France (2012-2017).

25