<figure><img src="image_1.png" /></figure>

HAL
open science

# Hyperspectral Sharpening Approaches Using Satellite Multiplatform Data

Rocco Restaino, Gemine Vivone, Paolo Addesso, Jocelyn Chanussot

<figure><img src="image_2.png" /></figure>

## To cite this version:

Rocco Restaino, Gemine Vivone, Paolo Addesso, Jocelyn Chanussot. Hyperspectral Sharpening Approaches Using Satellite Multiplatform Data. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (1), pp.578-596. 10.1109/TGRS.2020.3000267. hal-03429685

HAL Id: hal-03429685

https://hal.science/hal-03429685v1

Submitted on 26 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Hyperspectral Sharpening Approaches Using Satellite Multiplatform Data

Rocco Restaino<sup>[ID]</sup>, Senior Member, IEEE, Gemine Vivone<sup>[ID]</sup>, Senior Member, IEEE, Paolo Addesso<sup>[ID]</sup>, Member, IEEE, and Jocelyn Chanussot<sup>[ID]</sup>, Fellow, IEEE

**Abstract**—The use of hyperspectral (HS) data is growing over the years, thanks to the very high spectral resolution. However, HS data are still characterized by a spatial resolution that is too low for several applications, thus motivating the design of fusion techniques aimed to sharpen HS images with high spatial resolution data. To reach a significant resolution enhancement, high-resolution images should be acquired by different satellite platforms. In this article, we highlight the pros and cons of employing real multiplatform data, using the EO-1 satellite as an exemplary case. The spatial resolution of the HS data collected by the Hyperion sensor is improved by exploiting both the ALI panchromatic image collected from the same platform and acquisitions from the WorldView-3 and the QuickBird satellites. Furthermore, we tackle the problem of assessing the final quality of the fused product at the nominal resolution, which presents further difficulties in this general environment. Useful indications for the design of an effective sharpening method in this case are finally outlined.

**Index Terms**—Hyperion data, hyperspectral (HS) sharpening, image fusion, pansharpening, remote sensing.

## I. INTRODUCTION

HYPERSPECTRAL (HS) data represent a fundamental class of remotely sensed signals that are employed in many different fields such as, for instance, vegetation monitoring, water resource management, soil investigation, and geology [1], [2]. The use of HS images has been motivated by their superior capacity of discriminating key features of the illuminated scene with respect to multispectral (MS) images, as testified by several studies [3], [4]. However, the exploitation of HS data acquired by sensors on board of satellites is strongly limited nowadays because of the low spatial resolution (ranging from 30 m to 10 m). Most applications experiment a significant performance degradation due to the coarse ground spatial interval (GSI). Some instances are soil composition estimation [5],

Manuscript received February 1, 2020; revised April 3, 2020 and May 8, 2020; accepted May 30, 2020. (Corresponding author: Rocco Restaino.)

Rocco Restaino and Paolo Addesso are with the Department of Information Engineering, Electrical Engineering and Applied Mathematics, University of Salerno, 84084 Fisciano, Italy (e-mail: restaino@unisa.it; paddesso@unisa.it).

Gemine Vivone is with the Institute of Methodologies for Environmental Analysis, CNR-IMAA, 85050 Tito Scalo, Italy, and also with the Department of Information Engineering, Electrical Engineering and Applied Mathematics, University of Salerno, 84084 Fisciano, Italy (e-mail: gemine.vivone@imaa.cnr.it; gvivone@unisa.it).

Jocelyn Chanussot is with Université Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France (e-mail: jocelyn.chanussot@gipsa-lab.grenoble-inp.fr).

Color versions of one or more of the figures in this article are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TGRS.2020.3000267

vegetation classification [6], and urban change detection [7]. On the other hand, investigations using simulated and real data have demonstrated benefits by improving the spatial resolution through data fusion; see, for example, the studies on the estimation of the leaf area index [8], mineral mapping [9], and invasive plants detection [10]. These results have motivated the development of a large number of techniques for combining HS images with high spatial resolution (HR) data, acquired by MS and panchromatic (PAN) sensors. However, the sole decommissioned Earth observing (EO)-1 satellite mounted an instrument (advanced land imager - ALI) that is able to acquire on board both MS and PAN images together with HS data; moreover, only the panchromatic camera (characterized by a spatial resolution of 10 m) is exploitable to enhance the spatial resolution of the HS image acquired by the Hyperion HS sensor (featuring a 30 m GSI). Among the new and forthcoming satellites, Prisma, whose HS sensor has a 30 m GSI, and HypXIM (with 8 m HS GSI) will also mount a panchromatic camera (5 and 2 m GSIs, respectively) to enhance the HS spatial resolution [2].

Thus, a valuable option can be provided by fusing data from sensors on board of different platforms. However, multiplatform data fusion introduces further issues regarding, for example, data calibration and, more critically, the consistency of the images to fuse. In fact, crucial problems concern the different point of view and the different acquisition time that can be hardly compensated through the remotely sensed data-processing chain. On the contrary, most of this articles facing the problem of HS image sharpening only include tests performed on simulated data, which ignore the described practical issues. Even in the comprehensive reviews [11], [12], the algorithms were only assessed on simulated data. Only few instances can be found in literature addressing the fusion of real data, but working with degraded data [13]–[15]. A first attempt of fusing real data acquired by sensors on board of drones and aircraft both at degraded and nominal resolutions has been recently proposed, but only addressing the simpler case of HS pansharpening with no temporal misalignment among data [16].

### A. Background and Related Works

The goal of this article is to fill this gap by providing an extensive comparison of the fusion algorithms on real data collected by both single and multiple platforms at different acquisition times. The literature related to the enhancement
of HS images is broad [11], [12]. The most direct approach consists in using the techniques developed for the fusion of MS and PAN data (pansharpening) [17]. This methodology has the merit of leveraging on well-known algorithms, whose performance has been deeply investigated [18], [19], and thus constituting a yardstick for all the newly developed techniques, both in terms of image quality and computational burden. The application of pansharpening methods to HS/PAN fusion (usually called HS pansharpening) is straightforward and can be helped by a preliminary dimensionality reduction step [20], [21]; on the contrary, the generalization to HS/MS fusion requires additional work. A successful solution consists in assigning to each HS band, whose enhancement is separately performed, a single channel of the MS data [22], [23]. In general, superior performance can be achieved by removing the constraint of the use of a unique MS channel. Indeed, a reference method is constituted by hypersharpening that utilizes a weighted combination of all the MS bands for the spatial improvement of each HS band [24], [25].

Specific approaches developed for sharpening HS images are populating the literature. Most of them are based on the Bayesian statistical formalization of the problem describing the available observed data, namely the low spatial resolution (LR) HS, HR PAN, and HR MS images, as degraded versions of the unknown HR HS image [26]. Regularization techniques have been applied to overcome the ill-posed nature of such inverse problem. The origin of the penalization term can be due to a naive assignation of statistical models to the images [13], [26], a sparsity hypothesis [27], a quantification of the image vector total variation (VTV) (see the HySure method [28]), a model for the spectral features of the illuminated area (used in stochastic mixing model [29], and coupled nonnegative matrix factorization (CNMF) [30]).

An innovative research field is the application of the convolutional neural networks (CNN) to the spatial enhancement of optical images [31]. The fusion task is demanded to simple or complex neural network architectures, which are borrowed by other applications (as for example, super-resolution), or properly developed for the problem at hand. Flat networks, in which a single path can be individuated between the input and the output of the network, have been used in [32] and [33]. On the contrary, parallel processing by multiple substructures have been employed in component networks. For instance, different paths are followed by the original HS and MS data in the architecture proposed in [15], with the purpose of separately processing the spectral and the spatial information. The performance achievable by the flat and component networks are compared in [34].

A key problem of HS image enhancement is the assessment of fused products because of the lack of a reference image. A typical solution adopted for pansharpening relies upon the application of two protocols that leverage on complementary features [35]. In particular, a reduced resolution (RR) assessment is performed aiming at reproducing at a lower spatial resolution the characteristics of the addressed fusion problem, with the original HS image playing the role of the ground-truth (GT). Moreover, the full resolution (FR) assessment is also considered, in which the available data are

fused at the original resolution, thus renouncing to a target image. Obviously, the RR assessment introduces the further difficulty of performing a consistent image degradation phase, but has the merit of allowing the use of a wide class of quality indexes, which allow to accurately compare the fused product with the GT. On the other hand, the FR assessment does not require the spatial degradation of the available images, but the design of reliable quality indexes represents a hard task.

## B. Contribution

In this article, we focus on the sharpening of HS data acquired by the Hyperion sensor. We compare the quality of the final products generated by several fusion algorithms using either the panchromatic images collected by the ALI sensor mounted on the same platform, or one of the PAN and MS images acquired by a different platform. Two satellites acquiring PAN and MS data of the same area are used: QuickBird, which collects a PAN image and a 4-channel MS image, and WorldView-3, which collects a PAN image and a 8-channel MS image. In the latter case, a specific study on the effects of different acquisition times is presented. The compared algorithms have been chosen among effective classical approaches designed for pansharpening applications [35] and methods properly designed for HS image enhancement [24], [30].

To date, the evaluation of the quality of sharpened HS images has been mainly carried out at RR (see the review articles [11], [12] and the related references). In this work, we also performed the fusion of the images at FR, proposing an approach for evaluating the quality of the image in a challenging environment where multiple bands and sensors could be selected to sharpen a particular HS spectral band. Similar to most of the FR quality indexes, we disjointedly assess the spectral and the spatial fidelity of the fused product constructing an overall quality index by combining both the terms. The spectral distortion can be safely estimated by resorting to the consistency property of Wald's protocol [36], namely by comparing the degraded version of the final product to the original LR image [35], [37]. On the contrary, the spatial distortion can be only appreciated at the original resolution requiring the identification of a reference image. While in the classical PAN/MS fusion problem, there is no choice (only the PAN image is characterized by the target resolution), multiple choices are possible in a more general fusion problem. For instance, in our case, multiple high-resolution MS and PAN images are available. Previous studies illustrated how the details present in the scene depends on the specific channel, being strongly related to the portion of the electromagnetic spectrum reflected by the illuminated scene [38]. For this reason, the HR reference image should be collected by a sensor with similar spectral responses with respect to the ones of the sensor to enhance. Accordingly, the spatial assessment procedure that we designed in this article consists in selecting, for each LR channel to enhance, the HR image acquired by the sensor with the most similar relative spectral response (RSR). The RSR expresses the capacity of the sensor of acquiring energy as a function of the wavelength, namely, it represents,
on average over the illuminated scenes, the distribution of the collected energy on the frequency axis. The consequent analogy with the probability density function of random variables motivates the adoption of a statistical distance for comparing the RSRs [39], [40]. However, since the goal is to compare the spatial content in a given frequency range, a suitable metric should also take into account the distance between the frequency supports of the RSRs. The Wasserstein distance [41], which has a broad applicability in signal processing and machine learning [42], fits this requirement [43], [44]. Its minimization is indeed equivalent to find the solution of the optimal transport problem [45], which constitutes an outstanding formalization for many key image-processing issues, as, for example, resolution change [46], image registration [47], image retrieval [48], image equalization [49], color transfer [50], and segmentation [51].

The main contributions of this article can be summed up as follows.

1) A detailed analysis about the capabilities of several HS sharpening approaches to fuse real data acquired by single and multiple satellite platforms, even considering the crucial temporal misalignment factor, is proposed for the first time in the literature.

2) A complete FR assessment, complementing the RR validation, is proposed. A protocol to evaluate the spatial quality of the fused products at FR in a general multisource data fusion problem has been designed.

3) A quantitative and qualitative comparison of the products achievable using single and multiplatform settings is carried out for a wide range of resolution enhancement factors, obtaining useful indications for choosing fusion algorithms to be employed in the different cases.

### C. Organization

The remaining of this article is organized as follows. In Section II, we illustrate the problem of sharpening HS data detailing the employed mathematical notation. The specific features of the main image fusion algorithms are also pointed out. Section III deals with the critical assessment phase, with particular reference to the problem of the quality evaluation of final products at nominal resolution. The experimental phase is described in Section IV, while the quantitative and qualitative results are reported and discussed in Section V. Finally, conclusions are drawn in Section VI.

## II. HYPERSPECTRAL SHARPENING FRAMEWORK

Borrowing the definitions from the seminal article [36], the objective of satellite resolution enhancement through data fusion is to obtain, starting from an image collected by a low spatial resolution (LR) sensor, the image that the sensor would acquire if it had a higher (target) spatial resolution. To obtain the desired product, we make use of high spatial resolution (HR) companion images composed by a single or by multiple channels, namely, panchromatic (PAN) or MS images.

Thereon, we will denote a generic multichannel image as $A = \{A_k\}_{k=1,...,N_b}$, in which $A_k$ is the kth channel and $N_b$ is the number of bands. More specifically, we indicate the

LR HS image as $H = \{H_k\}_{k=1,...,N_h}$, the HR MS image as $M = \{M_m\}_{m=1,...,N_M}$, and the HR PAN image as $P = \{P_m\}_{m=1}$.

The ratio between the spatial resolution of the original HS image and the target resolution is the enhancement resolution ratio and will be denoted by $R$.

$$ \hat{\mathbf{H}} = \Psi(\mathbf{H}, C_R) \qquad (1) $$

in which $\Psi$ is an, eventually nonlinear, combination function and $C_R$ is the set of all the available images at target resolution. However, we restrict here the analysis to simple fusion schemes employing a unique HR image with the aim of drawing indications for real applications.

We mainly consider computationally efficient approaches selected among classical linear pansharpening methods that are resumed by the formula [35]

$$ \hat{\mathbf{H}}_k = \tilde{\mathbf{H}}_k + \mathbf{G}_k \circ (\mathbf{Y}_k - \mathbf{Y}_k^{LP}) \qquad (2) $$

in which $\circ$ is the element-by-element matrix product, and for each HS channel $k \in \{1, ..., N_H\}$, $\mathbf{G}_k$ is the injection gain matrix, $\tilde{\mathbf{H}}_k$ is the $\mathbf{H}_k$ image upsampled to the target resolution, $\mathbf{Y}_k$ is the employed HR image and $\mathbf{Y}_k^{LP}$ is its low pass version. The general expression (2) formalizes the different classical pansharpening approaches according to the particular injection matrix $\mathbf{G}$ and to the specific approach used to generate $\mathbf{Y}^{LP}$.

Pansharpening approaches, in which $\mathbf{Y}_k^{LP}$ is obtained as a combination of the $\tilde{\mathbf{H}}$ channels, are named component substitution (CS), whereas if $\mathbf{Y}_k^{LP}$ is obtained by spatially degrading $\mathbf{Y}_k$, we have the so-called multiresolution analysis (MRA) methods. Indeed, the former ones can be recast as the substitution of the intensity component of the HS image in a transformed domain, generated, for example, through the intensity-hue-saturation (IHS) [52], [53], the principal component decomposition (PCS) [54]–[56], or the Gram-Schmidt (GS) expansion [57]. For this reason, they are also called spectral methods, in opposition to spatial methods that is an alternative name of MRA approaches, which use multiple spatial filters for completing the multiscale decomposition. Box [58], [59], Gaussian [60] and conjugate mirror filters [61] are classical examples of analysis filters used for extracting the detail component of the HR image $\mathbf{Y}_k$ to be injected into $\tilde{\mathbf{H}}_k$. Crucially, the two classes are distinguished by complementary features in terms of robustness. In fact, CS methods are less sensitive to spatial misalignments and aliasing errors [62], whereas MRA approaches are more prone to compensate spatial inconsistencies of the fused images that are typically due to temporal changes of the illuminated scene or to different acquisition features of the instruments [63].

According to (2), classical pansharpening algorithms independently deal with the kth HS channel in conjunction to the associated $\mathbf{Y}_k$ and $\mathbf{Y}_k^{LP}$ images, which can vary significantly over $k$ also in HS/PAN fusion, due to the preliminary equalization step [17]. In the case of HS/MS fusion, the construction of a suitable single channel image to use in (2) is more critical, since multiple HR channels are available. This issue can be solved in two distinct manners, which are illustrated
<figure><img src="image_4.png" /><figcaption>Fig. 1. Fusion of HS and MS images through classical algorithms. (a) Application of pansharpening techniques to groups composed by a channel of the MS image and a group of HS channels. (b) Band-by-band sharpening of HS image through synthesized band images. (a) Band selection. (b) PAN synthesis.</figcaption></figure>

by Fig. 1: 1) a specific MS channel is selected through an assignment procedure, see Fig. 1(a); 2) a synthetic HR image is constructed starting from all the available MS channels, see Fig. 1(b). The performance achievable by adopting the first approach depends on the particular assignment method. Top approaches consist in maximizing the cross correlation (CC) between the MS channel and the specific $\mathbf{H}_k$ [22], or in choosing the MS channel that produces the minimum spectral distortion when substituted to $\mathbf{H}_k$ [23]. Both these methods have a band assignment that is data set-dependent. An alternative approach, that was proven to yield appreciable results [23] although its output depends only on the characteristics of the acquisition sensors, is the minimization of the distance between the centroids of the RSRs [64] of $\mathbf{H}_k$ and of the various MS channels. Denoting by $\rho_{A_i}(\lambda)$ the RSR of the sensor acquiring a generic channel $\mathbf{A}_i$ normalized to have unit integral over $\mathbb{R}$, the centroid of $\rho_{A_i}(f)$ is given by

$$ \mu_{A_i} = \int \rho_{A_i}(\lambda) \lambda \, d\lambda. \quad (3) $$

Then, $\mathbf{H}_k$ is assigned to the MS band $\mathbf{M}_n$ with the closest centroid, namely,

$$ \mathbf{M}_n = \operatorname{argmin}_{\mathbf{M}_j} |\mu_{\mathbf{H}_k} - \mu_{\mathbf{M}_j}|. \quad (4) $$

Hypersharpening through synthesized band [24], [25] instead adopts approach ii), i.e., a synthetic HR image $\mathbf{V}_k$ is generated for each $\mathbf{H}_k$ according to the formula

$$ \mathbf{V}_k = \sum_{m=1}^{N_M} w_{k,m} \mathbf{M}_m \quad (5) $$

where the weights $w_{k,m}$ are computed by minimizing the mean square error (MSE) between the RR version of $\mathbf{V}_k$ and $\tilde{\mathbf{H}}_k$. The

fusion formula is thus given by

$$ \tilde{\mathbf{H}}_k = \tilde{\mathbf{H}}_k + \frac{\operatorname{cov}(\tilde{\mathbf{H}}_k, \tilde{\mathbf{V}}_k^d)}{\operatorname{cov}(\tilde{\mathbf{V}}_k^d, \tilde{\mathbf{V}}_k^d)} (\mathbf{V}_k - \tilde{\mathbf{V}}_k^d) \quad (6) $$

in which $\operatorname{cov}(\cdot, \cdot)$ indicates the covariance operator and $\tilde{\mathbf{V}}_k^d$ is the interpolated (by the factor $R$) version of the decimated version $\mathbf{V}_k^d$ of $\mathbf{V}_k$. Improved performance can be obtained by substituting $\mathbf{V}_k$ with its equalized version, achieved by matching its first two moments with those of $\tilde{\mathbf{H}}_k$ [25].

Bayesian approaches constitute a widely employed alternative class of algorithms. In this formalization, the available LR and HR images are seen as degraded versions of the target HR HS image, say $\mathbf{Z}$ [11]. Using the lowercase letter to denote the columnized version of those images, in which each row contains all the pixel related to a single channel, the following equations are used

$$ \mathbf{h} = \mathbf{zBS} + \mathbf{n}_H \quad (7) $$

$$ \mathbf{m} = \mathbf{R}_M \mathbf{z} + \mathbf{n}_M \quad (8) $$

$$ \mathbf{p} = \mathbf{R}_P \mathbf{z} + \mathbf{n}_P \quad (9) $$

where (8) and (9) are employed for HS/MS and HS/PAN fusion, respectively. In the previous formulas, $\mathbf{B}$ is the blurring operator, written as a block circulant with circulant blocks (BCCB) matrix [65], $\mathbf{S}$ is the downsampling matrix, $\mathbf{R}_M$ and $\mathbf{R}_P$ are the matrices that take into account the RSRs of the MS and PAN channels, respectively, and $\mathbf{n}_H$, $\mathbf{n}_M$, and $\mathbf{n}_P$ are the columnized versions of the HS, the MS, and the PAN noises, which model the error committed using the sole first term.

Spectral unmixing [66] is commonly employed to ease the problem of recovering $\mathbf{z}$ from $\mathbf{h}$, and $\mathbf{m}$ or $\mathbf{p}$. In this approach, illustrated by Fig. 2, the pixel value is supposed to
<figure><img src="image_5.png" /><figcaption>Fig. 2. Fusion of HS and MS images through Bayesian algorithms exploiting spectral unmixing.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 3. Illustration of the reference image assignation problem. The RSRs of a LR channel (in black continuous line) and three exemplary RSRs of HR channels (in red, blue, and green discontinuous lines) are shown in the figure.</figcaption></figure>

be obtainable as a linear combination of endmembers, each modeling a specific feature present in the scene. Thus, **z** assumes the factorized form

$$z = EU \qquad (10)$$

where **E** is the spectral signature matrix and **U** the abundance matrix.

The problem of estimating the unknown HR HS image **z** is thus solved by maximizing the posterior probability of **z**, given the observations **h** and **m** or **p**

$$\hat{z} = \arg\max_{z} p(z|\mathbf{h}, \mathbf{m}, \mathbf{p}). \qquad (11)$$

Different ways have been adopted to regularize the inverse problem (11): examples are represented by the naive assignment of a Gaussian prior distribution to the coefficients **U** [67] and by the construction of a penalization term based on the sparse representation theory [68] or vector total variation (VTV) [28].

### III. FULL RESOLUTION ASSIGNMENT PROBLEM

Since a reference image at FR is unavailable, the FR assessment of sharpened products has to rely upon the original images. Usually, the final product quality is separately evaluated in the spectral and spatial domains, through a spectral distortion index $D_λ$ and a spatial distortion index $D_S$, respectively, and the two measures are then combined. Noticeable examples are the *quality w/ no reference* (QNR) index [69], *Khan's protocol* [37], the *hybrid quality w/ no reference* index (HQNR) [70], and the QNR+ index [71]. The spectral distortion is computed, in accordance to Wald's consistency property [36], using as a reference the original low resolution data. On the contrary, the reference data suitable for

assessing the spatial quality of the fused image are constituted by all the images available at target resolution, whose number depends on the specific problem. Accordingly, the choice of the most suitable reference image is still an open issue that we face in this study.

To preserve the features of the original low resolution sensor, physical considerations force to evaluate the spatial quality in comparison to a HR image that contains the details in the same frequency range [38]. Accordingly, we solved the FR assessment problem by associating, to each LR channel, the HR image collected by the sensor with the most similar spectral response. This assures also that the association does not depend on the features of the specific scenario, thus leading to consistent results across data sets. The purpose of this section is to illustrate the rationale behind the choice of a suitable similarity measure among RSRs.

To evaluate the difference from the ideal situation in which the RSRs of the LR and the reference channel coincide, an appropriate criterion should account both for the likeness of frequency components present in the RSR of the LR channel and for the absence of frequency components that are not included in the LR channel response. With reference to Fig. 3, in which the RSR of an LR channel is depicted in black continuous line and three RSRs of possible reference channels are depicted in blue, red, and green discontinuous lines, respectively, we expect that the best association is obtained using the channel with the red RSR. In fact, also if the support of the HR 3 channel includes that of the LR channel and the two RSR centroids are very close to each other, the wide bandwidth leads to some details that cannot be perceivable by the LR sensor. Furthermore, although both the supports of channels HR 1 and HR 2 are disjoint from that of the LR channel, we expect that the details collected by the HR 2 sensor more closely match those acquired by the LR sensor.

It is clear that we are ignoring at first glance the problem of the temporal coherence between data to be fused. Indeed, we have a coherent set of data when the single platform setting
is exploited. Instead, a certain incoherence between the LR HS and the HR MS/PAN images, depending on many factors, such as target spatial resolution and differences in the acquisition time and captured landscape, can be remarked in the case of multiplatform data fusion. This further issue will be addressed in Section IV-A by selecting a proper quality index to measure the spatial distortion of the fused products, even considering the possible temporal misalignment between data to be fused.

To perform the reference image assignation, the spectral responses are firstly normalized to have unit integral, for taking into account the conversion process leading from the power collected by the sensor to the pixel digital number [64]. Thus, there is a clear analogy between spectral responses and probability density functions that can be exploited, but paying particular attention to the peculiarities of the problem at hand. In fact, most distances between probability distributions (e.g., Hellinger, L2, total variation) can lead to meaningless results; for example, they assume the maximum value for densities with nonoverlapping supports, regardless of their position on the real axis.

For this reason, a closer analogy can be found with optimal transport problems, which consist in minimizing the effort required to transform a probability density $f(x)$ into another, say $g(y)$, under a specific cost function $c(x, y)$ that quantifies the cost of moving a unit mass from $x$ to $y$ [45]. If the cost function is the distance $||x - y||^p$, with $p \in [1, \infty)$, the optimal transport problem can be quantitatively formulated as the calculation of the *p*-Wasserstein distance [41]

$$W_p(f, g) = \left( \inf_{X \sim f(x), Y \sim g(y)} \mathbb{E}_{\pi(x,y)} [d(X, Y)^p] \right)^{1/p} \quad (12)$$

where the infimum is taken over all the joint densities $\pi(x, y)$ of $X$ and $Y$ with marginals $f(x)$ and $g(y)$, respectively. The Wasserstein distance assumes a very manageable expression when the densities are defined over the real line $\mathbb{R}$. Let us denote by $F(x)$ the cumulative distribution associated with the density $f(x)$ and by $F^{-1}(t)$ its (pseudo)-inverse

$$F^{-1}(t) = \inf\{x : F(x) \ge t\}, \quad t \in (0, 1). \quad (13)$$

An equivalent expression for $W_p(f, g)$ is thus given by [72]

$$W_p(f, g) = \left[ \int_0^1 |F^{-1}(t) - G^{-1}(t)|^p dt \right]^{\frac{1}{p}} \quad (14)$$

where $G^{-1}(t)$ is the (pseudo)-inverse of the cumulative distribution associated with $g(y)$.

Accordingly, we associate to the $k$th channel $\mathbf{H}_k$ the available HR image whose normalized RSR is closest to that of $\mathbf{H}_k$ in terms of 2-Wasserstein distance. More specifically we select the HR reference image of $\mathbf{H}_k$, say $\mathbf{I}(\mathbf{H}_k)$, according to the formula

$$\mathbf{I}(\mathbf{H}_k) = \arg \min_{C \in \mathcal{C}_R} W_2(\rho_C(\lambda), \rho_{\mathbf{H}_k}(\lambda)) \quad (15)$$

in which $C_R$ indicates the set of HR channels available at target resolution.

The choice of the 2-Wasserstein distance is due to two reasons. The 2-Wasserstein distance admits an equivalent simplified form in the presence of normal distributions, which

can be used to approximate the RSRs of the sensors. In this hypothesis, $W_2(f, g)$ is equal to [73]

$$W_2(f, g) = (|\mu_f - \mu_g|^2 + |\sigma_f^2 - \sigma_g^2|)^{1/2} \quad (16)$$

in which $\mu_f$ and $\mu_g$ are the expected values of distributions $F$ and $G$, respectively, and $\sigma_f^2$ and $\sigma_g^2$ are their variances. It is noted that although the Gaussian approximation of the RSRs is only accurate for HS channels, it represents a robust approach even for channels with a general shape leading to the same optimal assignment in all the cases examined in this work.

Moreover, if the RSR of the HS channel $\mathbf{H}_k$ is modeled as a degenerate distribution around its mean $\mu_{\mathbf{H}_k}$ (as a consequence of the very reduced bandwidth with respect to the MS/PAN bands), the expression of the 2-Wasserstein distance can be simplified as

$$W_p(\rho_C, \rho_{\mathbf{H}_k}) = (\mathbb{E}_{\rho_C(\lambda)}[|\lambda - \mu_{\mathbf{H}_k}|^2])^{1/2}. \quad (17)$$

In different words, the 2-Wasserstein distance assumes a straightforward physical interpretation, since it quantifies the dispersion of the RSR $\rho_C(\lambda)$ with respect to the center wavelength of the HS channel. Moreover, it often reproduces the intuitive behavior illustrated at the beginning of this section, ranking, for example, the HR channels shown in Fig. 3 in the following order: 1) HR 2; 2) HR 3; and 3) HR 1.

Finally, it is also useful to remark that the assignation algorithm based on the 2-Wasserstein distance has a computational effort comparable to other techniques based on the sensors' RSRs, as for example the widespread method that minimizes the distance between the RSR centroids [23].

## IV. EXPERIMENTAL ANALYSIS

The tests point at evaluating the quality of the fused products with different spatial resolution enhancement factors $R$, ranging from 3 to 12. More specifically, the available HS images, collected by the Hyperion sensor mounted on board of the EO-1 satellite, are characterized by a GSI of 30 m. In all the cases, the companion image acquired by the ALI PAN sensor, mounted on the same satellite, is used to test the performance obtainable through the single platform fusion approach. Since the ALI PAN is a 10-m resolution sensor, the original image can be used for a resolution enhancement with factor $R = 3$, while interpolated versions (here obtained using a cubic polynomial approximation of the ideal filter) have to be constructed for higher enhancement factors. The images acquired by the ALI MS sensor are not used in this work, since it is characterized by the same GSI of the Hyperion HS sensor. On the contrary, the spatial resolutions of the sensors mounted on the other used platforms are higher than the target resolution for all the considered resolution enhancement factors. For this reason, the images are purposely degraded to the target resolution by properly filtering them, that is using filters with amplitude responses that match the modulation transfer function (MTF) of the specific sensor, and by downsampling them.

Both the reduced and the FR procedures are employed in this work. In the first case, the available HS images are used as reference to compare the fused product, which, in turn,
<figure><img src="image_7.png" /><figcaption>Fig. 4. Reference image assignment based on the Wasserstein distance minimization. The RSR of the ALI PAN is plotted with a dashed line. (a) EO-1 & WV3 Fusion. (b) EO-1 & QB Fusion.</figcaption></figure>

is produced starting from degraded images. The latter are obtained according to Wald's protocol, namely, by reproducing the characteristics of the fusion problem at an RR. Even in this case, the degradation procedure has been carried out by filtering the original images through MTF-matched filters and a downsampler.

## A. Quality Indexes

The RR assessment protocol allows the use of many accurate quality indexes, since the GT image is available for the comparison. In this work, we consider the spectral angle mapper (SAM) [74] for evaluating the spectral distortion and the erreur relative globale adimensionnelle de synthèse (ERGAS) [75] for assessing the radiometric distortion. Finally, the vectorial Q2ⁿ-index [76] index is used for obtaining a comprehensive measure of the overall image quality.

For the FR assessment protocol, we adopt the recent QNR⁺ index [71] that is composed by a spectral distortion index Dᵢ and a spatial distortion index Dₛ.

The implementation of Dᵢ is borrowed from the consistency check proposed in [37]. It consists in comparing, by means of the Q2ⁿ index, the original MS image to the spatially degraded version of the fused product. The latter is obtained by means of a linear time-invariant filter with amplitude frequency response matching the MTF of the HS sensor, and a downsampler.

Instead, the spatial distortion index Dₛ is defined as

$$D_s = 1 - CD \qquad (18)$$

where CD is the coefficient of determination between the HR reference image and its approximation as linear combination of fused product channels. It is worth to be remarked that in the classical pansharpening, where data are simultaneously acquired and the HR reference image is temporally aligned to the fused product, this index is only able to measure the spatial distortion in the fused product. Instead, when we have data that are temporally misaligned, this index is also able to capture the distortion due to the injection of unexpected (temporally misaligned) HR details. Indeed, there is a direct impact of the temporal misalignment on the residuals of the regression between the HR reference image and the fused HS image affecting the value of the CD with its reduction and the

consequent increment of the related Dₛ value. In particular, MRA methods are penalized because they are not able to rearrange the HS low spatial frequencies to align them to the HR injected details [62], thus generating artifacts in the fused image accounted by the Dₛ index in (18). Instead, thanks to the ability of CS approaches to align the HS low spatial frequencies in agreement with the HR MS/PAN injected details [62], they still get high performance with a generally lower Dₛ index than MRA-based approaches, see, for instance, the FR results in Section V-B. But, this is strongly paid by a considerable increase of the spectral distortion, that is a higher value of the Dᵢ index, see again Section V-B, suggesting the exploitation of MRA-based approaches when temporal misalignments deeply affect data to be fused [63].

To extend (18) to the problem at hand, it is applied to each HR reference image by associating the HS channels according to the protocol detailed in Section III. Finally, the overall Dₛ index is simply obtained by averaging the Dₛ values related to each involved HR reference image. Fig. 4 reports the assignment of the HS bands to the available images. It is noted that the ALI PAN image (depicted with a dotted line) is available only for a resolution enhancement ratio R = 3.

The QNR⁺ is finally achieved from the spectral and the spatial distortion indexes according to the formula

$$QNR_{+} = (1 - D_{\lambda})^{\alpha}(1 - D_{S})^{\beta} \qquad (19)$$

in which α and β are two exponents aimed to balance the different variation experienced by the considered indexes. Suitable values of α and β can be estimated by fitting the QNR⁺ values to the Q2ⁿ indexes in an RR assessment environment exploiting a multivariate regression framework. The application of this procedure to all the data sets and fusion algorithms employed in this study has led to the values α = 1.5 and β = 0.5, which have been used to quantitatively assess the approaches at FR.

## B. Data Sets and Algorithms

In this article, we compare several fusion algorithms on four different test cases. The first three data sets include images acquired by the Hyperion and the WorldView-3 (WV3) sensors. The first group of data refers to the city of New York, while the other two images are representations of mixed areas
<figure><img src="image_8.png" /><figcaption>Fig. 5. RSRs of the Hyperion HS sensor (dotted lines), the WV3 and the QB sensors (both depicted by continuous lines). In the second row, the RSRs of the HS bands are painted with the same color of the MS band assigned by the assignment algorithm employed for the HS/MS fusion (i.e., the algorithm based on the minimization of the distance between the centroids). (a) Hyperion & ALI PAN. (b) Hyperion & QB PAN. (c) Hyperion & WV3 PAN. (d) Hyperion & QB MS. (e) Hyperion & WV3 MS.</figcaption></figure>

around the city Naples (Italy), differing for the time interval between the acquisitions of the two satellites. The fourth data set is composed by images of an urban area of San Francisco with the multiplatform data coming from the *QuickBird* (QB) sensor. Some further details about the exploited data sets are reported below:

1) *New York Data Set:* The images represent an urban area of New York, USA, alongside the Harlem river. The Hyperion and ALI data have been acquired on July 21, 2016 and the WV3 images on June 9, 2016. The visible near-infrared (VNIR) bands B09-B57 of the Hyperion data were used in the assessment and are characterized by a 15 bit radiometric resolution and a 144 × 72 image size. The size of the corresponding PAN ALI image is 432 × 216. The multiplatform data come from the WV3 satellite and they are composed by a PAN image and a 8-channel MS image (coastal, blue, green, yellow, red, red edge, NIR1, and NIR2), with sizes 14400 × 7200 pixels and 4320 × 2160 pixels, respectively, at the original resolution (0.3 m for the PAN sensor and 1.2 m for the MS sensor). The radiometric resolution is 11 bits.

2) *Agnano Data Set:* The images represent the West surroundings of Naples, Italy, around the Agnano Racecourse. The Hyperion and ALI data have been acquired on May 20, 2015 and the WV3 images on June 8, 2015. The characteristics of the acquired images are the same as for the New York data set.

3) *Capodichino Data Set:* The characteristics of the *Capodichino* data set, representing the zone around the Naples airport, are the same as those of the *Agnano* data set, except for the acquisition time of the WV3 images that dates back to February 4, 2002.

4) *San Francisco Data Set:* We used the B11-B57 channels of the Hyperion HS data and the ALI MS images collected on November 28, 2007. The companion multiplatform data were collected by the QB PAN and MS sensors on June 8, 2012. The MS sensor of the QB satellite acquires four bands (blue, green, red, and NIR) with a spatial resolution of 2.4 m and a panchromatic image with spatial resolution of 0.6 m. The radiometric resolution is 11 bits. The size of the HS data is 72 × 120 pixels, hence the PAN ALI image is composed by 216 × 360 pixels. The original size of the MS and PAN QB images related to the same area have size 900 × 1500 pixels and 3600 × 6000 pixels, respectively.

To evaluate the performance achievable through single platform and multiplatform data fusion in the described scenarios, we employ a number of classical sharpening algorithms, selected among both the CS and MRA classes, and fusion methods that were properly designed for HS sharpening. More specifically, the *Brovey transform (BT)* [77] and the *Gram Schmidt adaptive (GSA)* [78] are example of the CS class, while the *additive à trous wavelet transform (ATWT)* [19], the *smoothing filter-based intensity modulation (SFIM)* [58], [59], the *generalized Laplacian pyramid* [79] with MTF-matched filter [80] using both the *high-pass modulation* scheme [81] (*GLP-HPM*) and the *regression-based injection model (GLP-CBD)* [82] are prototypes of the MRA class. All the CS and MRA methods were implemented by using the assignation algorithm based on the minimization of the distance between centroids [23], resulting in the assignments shown in Fig. 5.

Furthermore, we test five approaches that were properly designed for the fusion of HS and MS data: the *hypersharpening (Hyper)*, proposed in [24] and optimized in [25] and
<caption>TABLE I</caption>

VALUES OF THE QUALITY INDEXES, RELATED TO THE RR ASSESSMENT PROCEDURE (LEFT) AND TO THE FR ASSESSMENT PROCEDURE (RIGHT), USING THE New York DATA SET, FOR RESOLUTION ENHANCEMENT RATIO $R = 3, 6, 12$. FOR EACH ALGORITHM, THE BEST RESULT AMONG THE HR OPTIONS IS IN BOLDFACE, WHILE THE BEST OVERALL RESULT IS HIGHLIGHTED IN BLUE

<table><thead><tr><th rowspan="3"></th><th colspan="3">ALI</th><th colspan="3">WV3</th><th colspan="3">ALI</th><th colspan="3">WV3</th></tr><tr><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">PAN</th></tr><tr><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th></tr><tr><th>Optimum</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>EXP</td><td>3.7283</td><td>5.9797</td><td>0.7186</td><td>3.7283</td><td>5.9797</td><td>0.7186</td><td>3.7283</td><td>5.9797</td><td>0.7186</td><td>0.0263</td><td>0.4948</td><td>0.6829</td><td>0.0263</td><td>0.4948</td><td>0.6829</td></tr><tr><td>BT</td><td>3.7283</td><td>5.6145</td><td>0.7938</td><td>3.7283</td><td>6.0073</td><td>0.7439</td><td><strong>3.4793</strong></td><td><strong>5.1871</strong></td><td><strong>0.8198</strong></td><td><strong>0.1184</strong></td><td>0.4140</td><td>0.6337</td><td>0.1695</td><td>0.1770</td><td>0.6867</td></tr><tr><td>GSA</td><td><strong>3.3517</strong></td><td><strong>4.1481</strong></td><td><strong>0.9081</strong></td><td>3.7548</td><td>5.6796</td><td>0.8356</td><td><strong>3.0279</strong></td><td><strong>4.6082</strong></td><td><strong>0.8491</strong></td><td>0.0210</td><td>0.4115</td><td>0.7431</td><td>0.0976</td><td>0.1267</td><td><strong>0.8011</strong></td></tr><tr><td>SFIM</td><td>3.6261</td><td>4.9713</td><td>0.8255</td><td>3.7693</td><td>5.4755</td><td>0.8032</td><td><strong>3.0279</strong></td><td><strong>4.6082</strong></td><td><strong>0.8491</strong></td><td>0.0210</td><td>0.4115</td><td>0.7431</td><td>0.0976</td><td>0.1267</td><td><strong>0.8011</strong></td></tr><tr><td>ATWT</td><td>3.5526</td><td>4.5694</td><td>0.9065</td><td>4.0865</td><td>6.4912</td><td>0.8288</td><td><strong>2.4580</strong></td><td><strong>3.6896</strong></td><td><strong>0.9247</strong></td><td><strong>0.0210</strong></td><td>0.4050</td><td>0.7472</td><td>0.0300</td><td>0.2025</td><td>0.8532</td></tr><tr><td>GLP-HPM</td><td>3.6117</td><td>4.4261</td><td>0.8863</td><td>3.7005</td><td>4.9314</td><td>0.8626</td><td><strong>2.4649</strong></td><td><strong>3.5097</strong></td><td><strong>0.9182</strong></td><td>0.0153</td><td>0.3948</td><td>0.7601</td><td>0.0170</td><td>0.2302</td><td>0.8551</td></tr><tr><td>GLP-CBD</td><td>3.3307</td><td>4.1032</td><td>0.9038</td><td>3.4904</td><td>5.0245</td><td>0.8560</td><td><strong>2.3504</strong></td><td><strong>3.6176</strong></td><td><strong>0.9225</strong></td><td><strong>0.0131</strong></td><td>0.3718</td><td>0.7771</td><td>0.0175</td><td>0.2005</td><td>0.8708</td></tr><tr><td>Hyper</td><td>3.3295</td><td>4.1032</td><td>0.9038</td><td>3.4902</td><td>5.0245</td><td>0.8560</td><td><strong>2.2103</strong></td><td><strong>4.0419</strong></td><td><strong>0.9053</strong></td><td><strong>0.0131</strong></td><td>0.3718</td><td>0.7771</td><td>0.0175</td><td>0.2005</td><td>0.8707</td></tr><tr><td>CNMF</td><td>3.7816</td><td><strong>4.5848</strong></td><td><strong>0.8758</strong></td><td>3.8235</td><td>6.0565</td><td>0.7883</td><td><strong>2.6989</strong></td><td><strong>4.8688</strong></td><td>0.8482</td><td><strong>0.0468</strong></td><td>0.3849</td><td>0.7299</td><td>0.1306</td><td>0.1074</td><td><strong>0.7658</strong></td></tr><tr><td>Bay-N</td><td>3.1806</td><td><strong>3.8555</strong></td><td><strong>0.9096</strong></td><td>3.2537</td><td>6.5074</td><td>0.7522</td><td><strong>2.4374</strong></td><td><strong>4.0418</strong></td><td>0.8944</td><td>0.0087</td><td>0.3912</td><td>0.7701</td><td><strong>0.0074</strong></td><td>0.2775</td><td>0.8406</td></tr><tr><td>Bay-S</td><td>3.0115</td><td><strong>3.7758</strong></td><td><strong>0.9124</strong></td><td>3.1228</td><td>6.2215</td><td>0.7664</td><td><strong>2.4351</strong></td><td><strong>4.0426</strong></td><td>0.8944</td><td>0.0087</td><td>0.3805</td><td>0.7769</td><td><strong>0.0073</strong></td><td>0.2703</td><td>0.8449</td></tr><tr><td>HySure</td><td>3.0462</td><td><strong>3.8992</strong></td><td><strong>0.9111</strong></td><td>6.0337</td><td>8.1958</td><td>0.7801</td><td><strong>2.2745</strong></td><td><strong>4.3912</strong></td><td>0.8874</td><td><strong>0.0232</strong></td><td>0.3702</td><td>0.7662</td><td>0.0551</td><td>0.2041</td><td>0.8194</td></tr><tr><td></td><td>5.3003</td><td>4.0747</td><td>0.4180</td><td>5.3003</td><td>4.0747</td><td>0.4180</td><td>5.3003</td><td>4.0747</td><td>0.4180</td><td>0.0301</td><td>0.6129</td><td>0.5942</td><td>0.0301</td><td>0.6129</td><td>0.5942</td></tr><tr><td>BT</td><td>5.3003</td><td>3.7601</td><td>0.5622</td><td>5.3003</td><td>3.5921</td><td>0.6174</td><td><strong>4.4974</strong></td><td><strong>3.2221</strong></td><td><strong>0.6699</strong></td><td><strong>0.1206</strong></td><td>0.4979</td><td>0.5843</td><td>0.1776</td><td>0.2395</td><td><strong>0.6504</strong></td></tr><tr><td>GSA</td><td>4.9412</td><td>3.3611</td><td>0.6682</td><td>4.8138</td><td>2.9234</td><td>0.7878</td><td><strong>3.2238</strong></td><td><strong>2.3240</strong></td><td><strong>0.8725</strong></td><td><strong>0.0280</strong></td><td>0.4460</td><td>0.7133</td><td>0.0931</td><td>0.1319</td><td><strong>0.8047</strong></td></tr><tr><td>SFIM</td><td>5.1786</td><td>3.5151</td><td>0.6039</td><td>5.0807</td><td>3.1865</td><td>0.7050</td><td><strong>4.0531</strong></td><td><strong>2.8766</strong></td><td><strong>0.7400</strong></td><td>0.0227</td><td>0.4930</td><td>0.6879</td><td>0.0242</td><td>0.3038</td><td>0.8042</td></tr><tr><td>ATWT</td><td>5.0903</td><td>3.2916</td><td>0.6850</td><td>5.0339</td><td>3.0008</td><td>0.8038</td><td><strong>3.4057</strong></td><td><strong>2.4014</strong></td><td><strong>0.8505</strong></td><td><strong>0.0183</strong></td><td>0.4738</td><td>0.7055</td><td>0.0230</td><td>0.2243</td><td>0.8506</td></tr><tr><td>GLP-HPM</td><td>5.1872</td><td>3.4123</td><td>0.6550</td><td>5.0419</td><td>3.0550</td><td>0.7425</td><td><strong>3.7534</strong></td><td><strong>2.6410</strong></td><td><strong>0.7841</strong></td><td>0.0195</td><td>0.4825</td><td>0.6984</td><td>0.0217</td><td>0.2568</td><td>0.8342</td></tr><tr><td>GLP-CBD</td><td>4.9043</td><td>3.2654</td><td>0.7070</td><td>4.7850</td><td>2.8550</td><td>0.8050</td><td><strong>3.1100</strong></td><td><strong>2.1966</strong></td><td><strong>0.8822</strong></td><td><strong>0.0174</strong></td><td>0.4549</td><td>0.7191</td><td>0.0227</td><td>0.2074</td><td>0.8602</td></tr><tr><td>Hyper</td><td>4.8965</td><td>3.2660</td><td>0.7066</td><td>4.7789</td><td>2.8560</td><td>0.8046</td><td><strong>2.5866</strong></td><td><strong>2.2494</strong></td><td><strong>0.8851</strong></td><td><strong>0.0174</strong></td><td>0.4549</td><td>0.7191</td><td>0.0227</td><td>0.2076</td><td>0.8601</td></tr><tr><td>CNMF</td><td>5.2895</td><td>3.3715</td><td>0.6926</td><td>5.0062</td><td>3.1229</td><td>0.7851</td><td><strong>3.1334</strong></td><td><strong>2.6798</strong></td><td><strong>0.8057</strong></td><td><strong>0.0487</strong></td><td>0.4544</td><td>0.6854</td><td>0.1404</td><td>0.1062</td><td>0.7535</td></tr><tr><td>Bay-N</td><td>5.5157</td><td>3.5327</td><td>0.6542</td><td>5.4466</td><td>3.3968</td><td>0.7583</td><td><strong>3.3950</strong></td><td><strong>2.5987</strong></td><td><strong>0.8498</strong></td><td><strong>0.0688</strong></td><td>0.4773</td><td>0.6496</td><td>0.0818</td><td>0.1811</td><td>0.7962</td></tr><tr><td>Bay-S</td><td>5.3729</td><td>3.5525</td><td>0.6365</td><td>5.3557</td><td>3.3342</td><td>0.7621</td><td><strong>3.4000</strong></td><td><strong>2.6005</strong></td><td><strong>0.8497</strong></td><td><strong>0.0697</strong></td><td>0.4725</td><td>0.6517</td><td>0.0802</td><td>0.1754</td><td>0.8011</td></tr><tr><td>HySure</td><td>6.0410</td><td>3.7730</td><td>0.6404</td><td>5.7569</td><td>3.5857</td><td>0.7407</td><td><strong>3.3605</strong></td><td><strong>2.6201</strong></td><td><strong>0.8492</strong></td><td><strong>0.0771</strong></td><td>0.4585</td><td>0.6524</td><td>0.1001</td><td>0.1561</td><td>0.7843</td></tr><tr><td></td><td>6.2367</td><td>2.3921</td><td>0.1220</td><td>6.2367</td><td>2.3921</td><td>0.1220</td><td>6.2367</td><td>2.3921</td><td>0.1220</td><td>0.0409</td><td>0.6935</td><td>0.5201</td><td>0.0409</td><td>0.6935</td><td>0.5201</td></tr><tr><td>BT</td><td>6.2367</td><td>2.2835</td><td>0.2615</td><td>6.2367</td><td>2.1695</td><td>0.3688</td><td><strong>5.4724</strong></td><td><strong>2.0232</strong></td><td><strong>0.4110</strong></td><td><strong>0.1279</strong></td><td>0.5872</td><td>0.5232</td><td>0.1977</td><td>0.3085</td><td><strong>0.5976</strong></td></tr><tr><td>GSA</td><td>6.1037</td><td>2.2278</td><td>0.2801</td><td>5.9646</td><td>1.9531</td><td>0.5006</td><td><strong>4.3934</strong></td><td><strong>1.5931</strong></td><td><strong>0.6879</strong></td><td><strong>0.0343</strong></td><td>0.5414</td><td>0.6427</td><td>0.0909</td><td>0.1560</td><td>0.7963</td></tr><tr><td>SFIM</td><td>6.1993</td><td>2.2418</td><td>0.2494</td><td>6.0699</td><td>2.0460</td><td>0.4052</td><td><strong>5.2477</strong></td><td><strong>1.9555</strong></td><td><strong>0.4306</strong></td><td>0.0329</td><td>0.5888</td><td>0.6096</td><td>0.0347</td><td>0.3397</td><td>0.7707</td></tr><tr><td>ATWT</td><td>6.2011</td><td>2.2261</td><td>0.2620</td><td>6.0610</td><td>1.8703</td><td>0.5863</td><td><strong>4.6656</strong></td><td><strong>1.6895</strong></td><td><strong>0.6268</strong></td><td><strong>0.0316</strong></td><td>0.5975</td><td>0.6046</td><td>0.0334</td><td>0.3016</td><td>0.7941</td></tr><tr><td>GLP-HPM</td><td>6.2410</td><td>2.1936</td><td>0.3117</td><td>6.0753</td><td>1.9944</td><td>0.4476</td><td><strong>5.1078</strong></td><td><strong>1.8857</strong></td><td><strong>0.4777</strong></td><td>0.0195</td><td>0.5715</td><td>0.6356</td><td>0.0230</td><td>0.2817</td><td>0.8184</td></tr><tr><td>GLP-CBD</td><td>6.0002</td><td>2.1125</td><td>0.4189</td><td>5.8815</td><td>1.7631</td><td>0.6456</td><td><strong>3.8261</strong></td><td><strong>1.3806</strong></td><td><strong>0.7824</strong></td><td><strong>0.0176</strong></td><td>0.5455</td><td>0.6564</td><td>0.0259</td><td>0.2078</td><td>0.8557</td></tr><tr><td>Hyper</td><td>5.9400</td><td>2.1156</td><td>0.4168</td><td>5.8315</td><td>1.7669</td><td>0.6426</td><td><strong>2.6696</strong></td><td><strong>1.1455</strong></td><td><strong>0.8740</strong></td><td><strong>0.0175</strong></td><td>0.5458</td><td>0.6563</td><td>0.0257</td><td>0.2084</td><td>0.8557</td></tr><tr><td>CNMF</td><td>6.2014</td><td>2.1308</td><td>0.4349</td><td>6.0337</td><td>1.6955</td><td>0.7837</td><td><strong>3.9529</strong></td><td><strong>1.3660</strong></td><td><strong>0.8214</strong></td><td><strong>0.0475</strong></td><td>0.5434</td><td>0.6282</td><td>0.1505</td><td>0.1269</td><td>0.7316</td></tr><tr><td>Bay-N</td><td>6.5074</td><td>2.2170</td><td>0.4120</td><td>6.3565</td><td>1.8564</td><td>0.7443</td><td><strong>3.2761</strong></td><td><strong>1.3585</strong></td><td><strong>0.8391</strong></td><td><strong>0.0652</strong></td><td>0.5616</td><td>0.5984</td><td>0.1217</td><td>0.1870</td><td>0.7423</td></tr><tr><td>Bay-S</td><td>6.5237</td><td>2.2229</td><td>0.4051</td><td>6.4374</td><td>1.8434</td><td>0.7474</td><td><strong>3.3089</strong></td><td><strong>1.3608</strong></td><td><strong>0.8387</strong></td><td><strong>0.0746</strong></td><td>0.5570</td><td>0.5925</td><td>0.1150</td><td>0.1814</td><td>0.7533</td></tr><tr><td>HySure</td><td>6.5074</td><td>2.1839</td><td>0.4323</td><td>6.3848</td><td>1.8194</td><td>0.7647</td><td><strong>3.5460</strong></td><td><strong>1.3661</strong></td><td><strong>0.8381</strong></td><td><strong>0.1102</strong></td><td>0.5705</td><td>0.5500</td><td>0.1245</td><td>0.1641</td><td>0.7490</td></tr></tbody></table>

Reduced Resolution

Full Resolution

four approaches based on the Bayesian formalization of the fusion problem: the *coupled nonnegative matrix factorization* (CNMF) [30], the *Naive Gaussian prior* (Bay-N) [67], the *Sparsity promoted Gaussian prior* (Bay-S) [68], and the *HS superresolution* (HySure) [28]. Finally, the method using the sole interpolation, based on a polynomial approximation of the ideal reconstruction filter, is employed as a yardstick and denoted as **EXP**.

## V. NUMERICAL RESULTS

An extensive numerical assessment campaign has been carried out. We analyzed the sharpened images produced in the four selected scenarios both from a quantitative and qualitative point of view. To this aim, the values of the quality indexes obtained using the fusion algorithms in the three settings are reported in Tables I–IV. In the first part of the analysis, we account for three data sets that are composed by multiplatform images representing very similar illuminated areas, thanks to a small time interval between the acquisition times or, equivalently, an almost static scenario. Afterward, we analyze the *Capodichino data set* that is composed by multiplatform data acquired at significantly different time instants referring to a more dynamic scenario (i.e., where rapidly changing objects can be noticed).

Close-ups of the final products obtained by four exemplary algorithms (GSA, SFIM, GLP-HPM and Hyper) are reported in Figs. 6–9. The included RGB images have been constructed using the red, green, and blue channels of the MS images and a mean of some channels in the red, green, and blue frequency ranges (B29-B33, B17-B22, and B11-B15, respectively) for the HS images.

## A. Performance Assessment With Temporally Aligned Data

Firstly, we consider the *New York* and the *Agnano* data sets that are illustrated by the close-ups in the first rows of Figs. 6 and 7. Among the investigated scenarios, they are the closest to the ideal situation, thanks to the small interval between the EO-1 and WV3 acquisition times. Accordingly, no significant changes can be noticed in the scene. This is the most common scenario in practical applications due to the fact that we have hundreds of satellites orbiting around the Earth every day. Thus, the problem of multiplatform image fusion with relative small temporal misalignments but a quite different acquisition geometry (i.e., the analysis at hand) is of great practical interest.

Afterward, we focus on the *San Francisco* data set that is composed by images shown in the first row of Fig. 8, which
<caption>TABLE II</caption>

VALUES OF THE QUALITY INDEXES, RELATED TO THE RR ASSESSMENT PROCEDURE (LEFT) AND TO THE FR ASSESSMENT PROCEDURE (RIGHT), USING THE *Agnano* DATA SET, FOR RESOLUTION ENHANCEMENT RATIO *R* = 3, 6, 12. FOR EACH ALGORITHM, THE BEST RESULT AMONG THE HR OPTIONS IS IN BOLDFACE, WHILE THE BEST OVERALL RESULT IS HIGHLIGHTED IN BLUE

<table><thead><tr><th rowspan="3"></th><th colspan="3">ALI</th><th colspan="3">WV3</th><th colspan="3">ALI</th><th colspan="3">WV3</th></tr><tr><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">MS</th><th colspan="3">PAN</th><th colspan="3">MS</th></tr><tr><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th></tr><tr><th>Optimum</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>EXP</td><td>2.6173</td><td>2.5563</td><td>0.8068</td><td>2.6173</td><td>2.5563</td><td>0.8068</td><td>2.6173</td><td>2.5563</td><td>0.8068</td><td>0.0151</td><td>0.4510</td><td>0.7242</td><td>0.0151</td><td>0.4510</td><td>0.7242</td></tr><tr><td>BT</td><td><b>2.6173</b></td><td>3.0548</td><td>0.8058</td><td>2.6173</td><td><b>2.8178</b></td><td>0.8049</td><td>3.2295</td><td>2.8406</td><td><b>0.8228</b></td><td>0.1142</td><td>0.4937</td><td>0.5932</td><td><b>0.1071</b></td><td>0.3065</td><td>0.7026</td></tr><tr><td>GSA</td><td><b>2.0386</b></td><td><b>1.9146</b></td><td><b>0.9131</b></td><td>2.6173</td><td><b>2.8178</b></td><td>0.8049</td><td>3.5162</td><td>3.3626</td><td><b>0.8386</b></td><td><b>0.0245</b></td><td>0.4248</td><td>0.7307</td><td>0.0707</td><td>0.2771</td><td>0.7617</td></tr><tr><td>SFIM</td><td>2.6038</td><td>2.3273</td><td>0.8550</td><td>2.6993</td><td>2.8019</td><td>0.8141</td><td><b>2.2512</b></td><td><b>2.0744</b></td><td><b>0.8897</b></td><td>0.0128</td><td>0.4138</td><td>0.7510</td><td>0.0129</td><td>0.3053</td><td>0.8175</td></tr><tr><td>ATWT</td><td>2.6542</td><td>2.5010</td><td>0.8875</td><td>2.9449</td><td>3.9281</td><td>0.7712</td><td><b>2.5303</b></td><td><b>2.3521</b></td><td><b>0.9076</b></td><td>0.0192</td><td>0.4146</td><td>0.7432</td><td>0.0284</td><td>0.2542</td><td>0.8270</td></tr><tr><td>GLP-HPM</td><td>2.6258</td><td>2.3637</td><td>0.8801</td><td>2.7184</td><td>2.8730</td><td>0.8351</td><td><b>2.0732</b></td><td><b>1.8683</b></td><td><b>0.9279</b></td><td>0.0133</td><td>0.4056</td><td>0.7557</td><td>0.0134</td><td>0.2407</td><td>0.8539</td></tr><tr><td>GLP-CBD</td><td><b>1.9534</b></td><td>1.8586</td><td>0.9140</td><td>3.3054</td><td>3.0199</td><td>0.8222</td><td>2.1162</td><td><b>1.9590</b></td><td><b>0.9251</b></td><td>0.0101</td><td>0.4088</td><td>0.7573</td><td>0.0128</td><td>0.2861</td><td>0.8288</td></tr><tr><td>Hyper</td><td>1.9543</td><td>1.8586</td><td>0.9140</td><td>3.3056</td><td>3.0199</td><td>0.8222</td><td><b>1.5896</b></td><td><b>1.7214</b></td><td><b>0.9382</b></td><td>0.0101</td><td>0.4088</td><td>0.7573</td><td>0.0128</td><td>0.2860</td><td>0.8288</td></tr><tr><td>CNMF</td><td>2.7780</td><td><b>2.6101</b></td><td><b>0.8581</b></td><td>2.6727</td><td>3.5643</td><td>0.7489</td><td><b>2.5088</b></td><td>2.7716</td><td>0.8443</td><td><b>0.0473</b></td><td>0.4665</td><td>0.6792</td><td>0.1027</td><td>0.2226</td><td>0.7494</td></tr><tr><td>Bay-N</td><td>2.2560</td><td><b>2.1131</b></td><td><b>0.9017</b></td><td>2.3423</td><td>3.5436</td><td>0.7494</td><td><b>2.2126</b></td><td>2.3448</td><td>0.8934</td><td>0.0072</td><td>0.4295</td><td>0.7472</td><td>0.0059</td><td>0.4345</td><td>0.7453</td></tr><tr><td>Bay-S</td><td><b>1.9847</b></td><td><b>1.8384</b></td><td><b>0.9192</b></td><td>2.2388</td><td>3.3907</td><td>0.7645</td><td>2.2131</td><td>2.3451</td><td>0.8934</td><td>0.0068</td><td>0.4106</td><td>0.7599</td><td><b>0.0056</b></td><td>0.4086</td><td>0.7625</td></tr><tr><td>HySure</td><td><b>2.3634</b></td><td><b>2.2301</b></td><td><b>0.8914</b></td><td>6.5977</td><td>6.0247</td><td>0.5739</td><td>2.5662</td><td>2.7840</td><td>0.8527</td><td><b>0.0271</b></td><td>0.4463</td><td>0.7141</td><td>0.1053</td><td>0.4433</td><td>0.6314</td></tr><tr><td>EXP</td><td>3.7601</td><td>1.7890</td><td>0.5523</td><td>3.7601</td><td>1.7890</td><td>0.5523</td><td>3.7601</td><td>1.7890</td><td>0.5523</td><td>0.0177</td><td>0.5693</td><td>0.6390</td><td>0.0177</td><td>0.5693</td><td>0.6390</td></tr><tr><td>BT</td><td>3.7601</td><td>1.8723</td><td>0.6009</td><td>3.7601</td><td>1.7321</td><td>0.6462</td><td><b>3.6738</b></td><td><b>1.5425</b></td><td><b>0.7624</b></td><td>0.1178</td><td>0.5781</td><td>0.5383</td><td><b>0.1102</b></td><td>0.3793</td><td>0.6612</td></tr><tr><td>GSA</td><td><b>2.9847</b></td><td><b>1.4490</b></td><td>0.7389</td><td>3.8581</td><td>1.6202</td><td>0.7664</td><td>3.4506</td><td>1.5113</td><td><b>0.8447</b></td><td><b>0.0269</b></td><td>0.5045</td><td>0.6758</td><td>0.0637</td><td>0.2767</td><td>0.7705</td></tr><tr><td>SFIM</td><td>3.7350</td><td>1.6602</td><td>0.6571</td><td>3.7439</td><td>1.6012</td><td>0.7261</td><td><b>2.9171</b></td><td><b>1.2709</b></td><td><b>0.8244</b></td><td>0.0151</td><td>0.4990</td><td>0.6919</td><td>0.0146</td><td>0.2909</td><td>0.8237</td></tr><tr><td>ATWT</td><td>3.7375</td><td>1.6493</td><td>0.6858</td><td>3.8327</td><td>1.7795</td><td>0.7466</td><td><b>3.1954</b></td><td><b>1.3865</b></td><td><b>0.8499</b></td><td>0.0148</td><td>0.4902</td><td>0.6982</td><td>0.0160</td><td>0.2553</td><td>0.8424</td></tr><tr><td>GLP-HPM</td><td>3.7500</td><td>1.6695</td><td>0.6946</td><td>3.7466</td><td>1.5902</td><td>0.7542</td><td><b>2.7382</b></td><td><b>1.1754</b></td><td><b>0.8645</b></td><td>0.0163</td><td>0.4907</td><td>0.6963</td><td>0.0155</td><td>0.2561</td><td>0.8425</td></tr><tr><td>GLP-CBD</td><td>2.9661</td><td>1.4434</td><td>0.7528</td><td>3.8682</td><td>1.6164</td><td>0.7758</td><td><b>2.8919</b></td><td><b>1.2444</b></td><td><b>0.8807</b></td><td>0.0136</td><td>0.4976</td><td>0.6943</td><td>0.0158</td><td>0.3044</td><td>0.8143</td></tr><tr><td>Hyper</td><td>2.9689</td><td>1.4419</td><td>0.7529</td><td>3.8674</td><td>1.6160</td><td>0.7758</td><td><b>2.0224</b></td><td><b>1.0446</b></td><td><b>0.9079</b></td><td>0.0136</td><td>0.4976</td><td>0.6944</td><td>0.0158</td><td>0.3046</td><td>0.8143</td></tr><tr><td>CNMF</td><td>3.8751</td><td>1.7322</td><td>0.7141</td><td>3.7664</td><td>1.8094</td><td>0.7379</td><td><b>2.6701</b></td><td><b>1.3872</b></td><td><b>0.8310</b></td><td><b>0.0519</b></td><td>0.5249</td><td>0.6363</td><td>0.1051</td><td>0.2153</td><td>0.7499</td></tr><tr><td>Bay-N</td><td>3.8832</td><td>1.7313</td><td>0.6794</td><td>4.0791</td><td>1.9624</td><td>0.6788</td><td><b>3.4757</b></td><td><b>1.6422</b></td><td><b>0.8013</b></td><td><b>0.0472</b></td><td>0.5288</td><td>0.6384</td><td>0.0557</td><td>0.3619</td><td>0.7330</td></tr><tr><td>Bay-S</td><td>3.7409</td><td>1.6761</td><td>0.6831</td><td>4.0403</td><td>1.9469</td><td>0.6797</td><td><b>3.4817</b></td><td><b>1.6433</b></td><td><b>0.8010</b></td><td><b>0.0468</b></td><td>0.5185</td><td>0.6457</td><td>0.0545</td><td>0.3396</td><td>0.7471</td></tr><tr><td>HySure</td><td>4.5009</td><td>2.0584</td><td>0.6151</td><td>4.1595</td><td>1.8173</td><td>0.6808</td><td><b>3.2202</b></td><td><b>1.5890</b></td><td><b>0.7744</b></td><td><b>0.0797</b></td><td>0.5301</td><td>0.6052</td><td>0.0772</td><td>0.3316</td><td>0.7247</td></tr><tr><td>EXP</td><td>4.6496</td><td>1.0807</td><td>0.2831</td><td>4.6496</td><td>1.0807</td><td>0.2831</td><td>4.6496</td><td>1.0807</td><td>0.2831</td><td>0.0285</td><td>0.6576</td><td>0.5603</td><td>0.0285</td><td>0.6576</td><td>0.5603</td></tr><tr><td>BT</td><td>4.6496</td><td>1.0984</td><td>0.3604</td><td>4.6496</td><td>1.0388</td><td>0.4331</td><td><b>4.2802</b></td><td><b>0.9126</b></td><td><b>0.6314</b></td><td><b>0.1291</b></td><td>0.4520</td><td>0.6017</td><td><b>0.1291</b></td><td>0.4520</td><td>0.6017</td></tr><tr><td>GSA</td><td>4.0761</td><td>0.9754</td><td>0.4544</td><td>4.1415</td><td>0.8962</td><td>0.6339</td><td><b>3.5818</b></td><td><b>0.7576</b></td><td><b>0.8114</b></td><td><b>0.0699</b></td><td>0.3190</td><td>0.7402</td><td><b>0.0699</b></td><td>0.3190</td><td>0.7402</td></tr><tr><td>SFIM</td><td>4.6484</td><td>1.0400</td><td>0.3980</td><td>4.6220</td><td>0.9544</td><td>0.5626</td><td><b>3.6373</b></td><td><b>0.7981</b></td><td><b>0.6792</b></td><td>0.0253</td><td>0.3380</td><td>0.7829</td><td>0.0253</td><td>0.3380</td><td>0.7829</td></tr><tr><td>ATWT</td><td>4.6406</td><td>1.0435</td><td>0.3731</td><td>4.6808</td><td>1.0071</td><td>0.6196</td><td><b>4.0778</b></td><td><b>0.8592</b></td><td><b>0.7334</b></td><td>0.0247</td><td>0.3351</td><td>0.7854</td><td>0.0247</td><td>0.3351</td><td>0.7854</td></tr><tr><td>GLP-HPM</td><td>4.6745</td><td>1.0314</td><td>0.4644</td><td>4.6288</td><td>0.9355</td><td>0.6071</td><td><b>3.4479</b></td><td><b>0.7399</b></td><td><b>0.7318</b></td><td>0.0169</td><td>0.2931</td><td>0.8195</td><td>0.0169</td><td>0.2931</td><td>0.8195</td></tr><tr><td>GLP-CBD</td><td>3.7656</td><td>0.9312</td><td>0.5485</td><td>4.6172</td><td>0.9707</td><td>0.7038</td><td><b>3.5286</b></td><td><b>0.7418</b></td><td><b>0.8378</b></td><td>0.0209</td><td>0.3403</td><td>0.7869</td><td>0.0209</td><td>0.3403</td><td>0.7869</td></tr><tr><td>Hyper</td><td>3.7727</td><td>0.9179</td><td>0.5485</td><td>4.5703</td><td>0.9647</td><td>0.7009</td><td><b>2.6346</b></td><td><b>0.6939</b></td><td><b>0.8396</b></td><td>0.0192</td><td>0.3404</td><td>0.7889</td><td>0.0192</td><td>0.3404</td><td>0.7889</td></tr><tr><td>CNMF</td><td>4.6379</td><td>1.0417</td><td>0.5240</td><td>4.5326</td><td>1.0233</td><td>0.6848</td><td><b>2.9578</b></td><td><b>0.7639</b></td><td><b>0.7985</b></td><td>0.1166</td><td>0.2661</td><td>0.7112</td><td><b>0.1134</b></td><td>0.2559</td><td>0.7202</td></tr><tr><td>Bay-N</td><td>5.0346</td><td>1.1319</td><td>0.5224</td><td>5.1672</td><td>1.1682</td><td>0.6518</td><td><b>3.9694</b></td><td><b>0.9266</b></td><td><b>0.7793</b></td><td><b>0.0977</b></td><td>0.3979</td><td>0.6651</td><td><b>0.0977</b></td><td>0.3979</td><td>0.6651</td></tr><tr><td>Bay-S</td><td>4.9414</td><td>1.1048</td><td>0.5230</td><td>5.1612</td><td>1.1716</td><td>0.6518</td><td><b>4.1740</b></td><td><b>0.9467</b></td><td><b>0.7705</b></td><td><b>0.0939</b></td><td>0.3833</td><td>0.6774</td><td>0.0940</td><td>0.3836</td><td>0.6771</td></tr><tr><td>HySure</td><td>5.3849</td><td>1.2276</td><td>0.5366</td><td>5.2701</td><td>1.1304</td><td>0.6484</td><td><b>3.2787</b></td><td><b>0.8430</b></td><td><b>0.7734</b></td><td>0.1125</td><td>0.3476</td><td>0.6754</td><td><b>0.1072</b></td><td>0.3575</td><td>0.6762</td></tr></tbody></table>

Reduced Resolution

Full Resolution

were collected by EO-1 and QB with a time delay of five years. However, minor effects of temporal variations are experienced, thanks to a mostly static landscape.

Finally, it is worth to be remarked that the different acquisition epochs of multiplatform data imply also a different contribution of the atmosphere on the at-sensor spectral radiance. In this problem, a key role is played by the path radiance that has to be taken into account by properly modeling the energy transmission channel [83], [84]. Although we do not tackle this issue in this work, we remark that most compared pansharpening methods automatically compensate for the path radiance effect.

1) *Reduced Resolution Assessment:* In the RR assessment, the quality indexes can be accurately computed with respect to the original HS image that acts as reference, or (GT). For this reason, all the employed measures are strongly consistent with the perceptible image quality. Thus, we solely analyze the quantities reported on the left side in Tables I–III that refer to the first three analyzed scenarios.

The outcomes related to the *New York* and *Agnano* data sets are practically the same. The first remarkable result is that almost all the algorithms achieve the best performance using the WV3 MS data, also when no interpolation is required to exploit the ALI PAN image (namely, for *R* = 3). This can

be justified by two different motivations. First, the frequency support of the ALI PAN overlaps only a small number of HS bands. However, this consideration is not sufficient to guarantee a performance improvement, as demonstrated by the comparison between the two HS/PAN fusion settings. In fact, despite the ALI PAN is characterized by a larger support, the use of the ALI PAN yields a superior quality, as a consequence of a greater spatial and temporal coherence. A second argument is thus required to explain the results. Indeed, the spatial information derived from the MS data is more specific, since it is related to a closer frequency band.

Focusing on the specific algorithms, the tests confirm that the CS and the Bayesian methods are more sensitive to misalignments. Actually, the sole GSA method achieves better values of the quality indexes by employing the ALI PAN, for both *R* = 3 and *R* = 6, while the Bayesian approaches obtain a superior quality only for *R* = 3. Moreover, a side product of the study is the ratification of the efficiency of the Hyper technique that obtains the best image quality over the whole set of algorithms.

The San Francisco data set has the purpose of corroborating the results when a different sensor acquiring the high spatial resolution images (i.e., the QB sensor) is exploited. The results have been computed for *R* = 3, 6 due to the smaller dimensions.
<caption>TABLE III</caption>

VALUES OF THE QUALITY INDEXES, RELATED TO THE RR ASSESSMENT PROCEDURE (LEFT) AND TO THE FR ASSESSMENT PROCEDURE (RIGHT), USING THE San Francisco DATA SET, FOR RESOLUTION ENHANCEMENT RATIO *R* = 3, 6. FOR EACH ALGORITHM, THE BEST RESULT AMONG THE HR OPTIONS IS IN BOLDFACE, WHILE THE BEST OVERALL RESULT IS HIGHLIGHTED IN BLUE

<table><thead><tr><th rowspan="3"></th><th colspan="3">ALI</th><th colspan="3">QB</th><th colspan="3">ALI</th><th colspan="3">QB</th></tr><tr><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">PAN</th></tr><tr><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th></tr><tr><th>Optimum</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>EXP</td><td>2.8718</td><td>3.8336</td><td>0.6366</td><td>2.8718</td><td>3.8336</td><td>0.6366</td><td>2.8718</td><td>3.8336</td><td>0.6366</td><td>0.0257</td><td>0.6389</td><td>0.5779</td><td>0.0257</td><td>0.6389</td><td>0.5779</td></tr><tr><td>BT</td><td>2.8718</td><td>4.6198</td><td><strong>0.7483</strong></td><td>2.8718</td><td>4.9768</td><td>0.5501</td><td><strong>2.7760</strong></td><td><strong>4.5698</strong></td><td>0.6724</td><td><strong>0.1271</strong></td><td>0.4792</td><td><strong>0.5885</strong></td><td>0.2665</td><td>0.1746</td><td>0.5707</td></tr><tr><td>GSA</td><td><strong>2.8907</strong></td><td><strong>3.6102</strong></td><td><strong>0.8014</strong></td><td>4.8465</td><td>7.4200</td><td>0.4805</td><td>4.1228</td><td>5.6438</td><td>0.6493</td><td><strong>0.0307</strong></td><td>0.5537</td><td><strong>0.6376</strong></td><td>0.2564</td><td>0.4280</td><td>0.4850</td></tr><tr><td>SFIM</td><td>2.9042</td><td>3.9822</td><td>0.7233</td><td>3.1814</td><td>5.1807</td><td>0.5712</td><td><strong>2.2508</strong></td><td><strong>3.4362</strong></td><td><strong>0.7701</strong></td><td><strong>0.0227</strong></td><td>0.5666</td><td><strong>0.6361</strong></td><td>0.0270</td><td><strong>0.4742</strong></td><td>0.6960</td></tr><tr><td>ATWT</td><td>3.1723</td><td>4.5905</td><td>0.7343</td><td>4.1302</td><td>7.7274</td><td>0.4639</td><td><strong>2.5497</strong></td><td><strong>3.4840</strong></td><td><strong>0.7754</strong></td><td><strong>0.0346</strong></td><td>0.5857</td><td>0.6106</td><td>0.0725</td><td>0.4263</td><td>0.6766</td></tr><tr><td>GLP-HPM</td><td>2.8748</td><td>3.9071</td><td>0.7715</td><td>3.2205</td><td>5.2800</td><td>0.5964</td><td><strong>2.1197</strong></td><td><strong>3.4058</strong></td><td><strong>0.8014</strong></td><td><strong>0.0216</strong></td><td>0.5584</td><td>0.6431</td><td>0.0278</td><td><strong>0.3978</strong></td><td>0.7439</td></tr><tr><td>GLP-CBD</td><td>2.8324</td><td>3.5507</td><td><strong>0.8025</strong></td><td>3.8616</td><td>5.5631</td><td>0.5819</td><td><strong>2.2165</strong></td><td><strong>3.4888</strong></td><td>0.7949</td><td><strong>0.0174</strong></td><td>0.5493</td><td>0.6539</td><td>0.0289</td><td>0.3978</td><td>0.7426</td></tr><tr><td>Hyper</td><td>2.8321</td><td><strong>3.5507</strong></td><td><strong>0.8025</strong></td><td>3.8639</td><td>5.5631</td><td>0.5819</td><td><strong>2.2352</strong></td><td>3.6448</td><td>0.7771</td><td><strong>0.0174</strong></td><td>0.5492</td><td>0.6539</td><td>0.0289</td><td>0.3978</td><td>0.7426</td></tr><tr><td>CNMF</td><td>2.9315</td><td>4.3280</td><td>0.7302</td><td>3.0798</td><td>4.7795</td><td>0.6058</td><td><strong>2.4770</strong></td><td><strong>3.9517</strong></td><td><strong>0.7433</strong></td><td><strong>0.0526</strong></td><td>0.4970</td><td>0.6540</td><td>0.1084</td><td>0.1635</td><td><strong>0.7700</strong></td></tr><tr><td>Bay-N</td><td>2.7969</td><td>4.1614</td><td>0.7342</td><td>2.7596</td><td>4.1074</td><td>0.6726</td><td><strong>2.2428</strong></td><td><strong>3.1433</strong></td><td><strong>0.8000</strong></td><td>0.0104</td><td>0.5217</td><td>0.6808</td><td><strong>0.0075</strong></td><td>0.4144</td><td>0.7566</td></tr><tr><td>Bay-S</td><td>2.6551</td><td>4.0073</td><td>0.7431</td><td>2.6916</td><td>4.0922</td><td>0.6735</td><td><strong>2.2390</strong></td><td><strong>3.1419</strong></td><td><strong>0.8002</strong></td><td>0.0102</td><td>0.4951</td><td>0.6996</td><td><strong>0.0077</strong></td><td>0.4149</td><td>0.7562</td></tr><tr><td>HySure</td><td><strong>2.8648</strong></td><td><strong>4.2242</strong></td><td><strong>0.7483</strong></td><td>7.1402</td><td>8.1669</td><td>0.3671</td><td>5.8092</td><td>6.2597</td><td>0.5161</td><td><strong>0.0366</strong></td><td>0.5071</td><td>0.6639</td><td>0.3551</td><td>0.5081</td><td>0.3632</td></tr><tr><td></td><td colspan="3">Reduced Resolution</td><td colspan="3">Full Resolution</td><td colspan="3"></td><td colspan="3"></td><td colspan="3"></td><td colspan="3"></td></tr></tbody></table>

sions of the available images. However, Table III indicates a behavior of the algorithms similar to the first two data sets. A favorable single platform data utilization is experienced for the approaches that extensively exploit the spatial information, namely for the CS approaches and for the **HySure** algorithm, mainly at *R* = 3, while in most cases, MRA and the remaining Bayesian algorithms are better fed by QB MS data. The results confirm that a significant improvement introduced by the utilization of the QB MS data is due to the possibility of properly enhancing Hyperion bands in the NIR range as arguable by comparing the results obtained in the preliminary study [14], wherein the sole HS bands in the visible range were considered.

2) Full Resolution Assessment: The same scenarios have been considered for the FR assessment phase. In this case, the qualitative analysis based on visual inspection plays a key role due to the minor reliability of the indexes, which use no reference image. In this study, the analysis of the final products reported in Figs. 6–8 is also useful to check the consistency of the proposed assessment protocol.

By analyzing the values referring to the RR assessment and those referring to the FR assessment, both reported in Tables I–III, the first key result is the evidence of a substantial match. Actually, the inadequacy of employing the CS, the CNMF, and the HySure methods with multiplatform data are still better highlighted by the FR protocol, since also for larger resolution enhancement factors, the GSA fails to obtain a significant performance improvement using data collected by the WV3 and QB satellites. Another element that is more noticeable in the FR quantitative assessment is the difficulty of obtaining a high spectral fidelity of the final product by multiplatform data. In fact, lower values of

the spectral distortion $D_\lambda$ are obtained using the WV3 data set only for the Agnano data set, while in the remaining tests, the fusion with ALI PAN data appears to be the best option.

The opposite behavior is experienced by the spatial distortion index, that is computed according to the protocol described in Section III. In the three considered cases, the choice of the reference HR images based on the Wasserstein distance leads to the assignments reported in Fig. 4. Reasonably, the reference image is always one of the MS bands of the WV3 or QB sensors that collects the electromagnetic radiation in a frequency range similar to the HS channel to enhance.

The appropriateness of the spectral and spatial distortion measures is corroborated by the visual analysis of the final products, see Figs. 6–8. The spatial distortion index is able to discriminate the accuracy of the details reproduction also among algorithms belonging to the same class, as it is evident by comparing the products obtained using the SFIM and GLP-HPM methods and the corresponding spatial distortion indexes.

All the data sets testify that significantly more appealing results are achievable through the multiplatform settings. This consideration matches the higher sensitivity of the human eye to the spatial details, which are better defined in the images obtained by employing accurately co-registered multiplatform high resolution data. Indeed, the availability of images at a resolution significantly higher than the target one (5 m for the considered images at *R* = 6) considerably eases this task, often permitting to obtain the desired alignment precision without recurring to additional interpolation steps, which always imply a degradation of the available data.
<caption>TABLE IV</caption>

VALUES OF THE QUALITY INDEXES, RELATED TO THE RR ASSESSMENT PROCEDURE (LEFT) AND TO THE FR ASSESSMENT PROCEDURE (RIGHT), USING THE Capodichino DATA SET, FOR RESOLUTION ENHANCEMENT RATIO $R = 3, 6, 12$. FOR EACH ALGORITHM, THE BEST RESULT AMONG THE HR OPTIONS IS IN BOLDFACE, WHILE THE BEST OVERALL RESULT IS HIGHLIGHTED IN BLUE

<table><thead><tr><th rowspan="3"></th><th colspan="3">ALI</th><th colspan="3">WV3</th><th colspan="3">ALI</th><th colspan="3">WV3</th></tr><tr><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">MS</th><th colspan="3">PAN</th><th colspan="3">MS</th></tr><tr><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th></tr><tr><th>Optimum</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>EXP</td><td>2.9041</td><td>3.4640</td><td>0.7607</td><td>2.9041</td><td>3.4640</td><td>0.7607</td><td>2.9041</td><td>3.4640</td><td>0.7607</td><td>0.0217</td><td>0.7237</td><td>0.5087</td><td>0.0217</td><td>0.7237</td><td>0.5087</td></tr><tr><td>BT</td><td><b>2.9041</b></td><td><b>4.1628</b></td><td><b>0.7804</b></td><td>2.9041</td><td>4.9941</td><td>0.6010</td><td>4.6792</td><td>5.7317</td><td>0.5938</td><td><b>0.1280</b></td><td>0.7666</td><td>0.3934</td><td>0.2951</td><td>0.3617</td><td><b>0.4729</b></td></tr><tr><td>GSA</td><td><b>2.4807</b></td><td><b>2.5976</b></td><td><b>0.8833</b></td><td>3.7976</td><td>5.4843</td><td>0.6316</td><td>7.0619</td><td>8.2274</td><td>0.5357</td><td><b>0.0228</b></td><td>0.7236</td><td><b>0.5079</b></td><td>0.3198</td><td>0.3274</td><td>0.4601</td></tr><tr><td>SFIM</td><td>2.9641</td><td>3.0793</td><td><b>0.8226</b></td><td>3.0186</td><td>3.7504</td><td>0.7493</td><td><b>2.6798</b></td><td>3.1641</td><td>0.8196</td><td><b>0.0174</b></td><td>0.7044</td><td>0.5296</td><td>0.0200</td><td>0.5790</td><td>0.6295</td></tr><tr><td>ATWT</td><td>3.1064</td><td>3.3752</td><td><b>0.8647</b></td><td>3.5159</td><td>5.3195</td><td>0.6730</td><td>3.4130</td><td>3.9316</td><td>0.7946</td><td><b>0.0199</b></td><td>0.7514</td><td>0.4837</td><td>0.0479</td><td>0.4705</td><td>0.6760</td></tr><tr><td>GLP-HPM</td><td>3.0965</td><td>3.1605</td><td><b>0.8526</b></td><td>3.0991</td><td>3.9384</td><td>0.7556</td><td><b>2.7198</b></td><td>3.1964</td><td>0.8373</td><td><b>0.0155</b></td><td>0.7256</td><td>0.5117</td><td>0.0236</td><td>0.5011</td><td>0.6814</td></tr><tr><td>GLP-CBD</td><td><b>2.4126</b></td><td><b>2.5037</b></td><td><b>0.8832</b></td><td>3.0741</td><td>3.4973</td><td>0.7781</td><td>2.6355</td><td>3.1271</td><td>0.8385</td><td><b>0.0122</b></td><td>0.7142</td><td>0.5249</td><td>0.0208</td><td>0.5118</td><td>0.6771</td></tr><tr><td>Hyper</td><td><b>2.4126</b></td><td><b>2.5037</b></td><td><b>0.8832</b></td><td>3.0739</td><td>3.4973</td><td>0.7781</td><td>2.7138</td><td>3.2570</td><td>0.8224</td><td><b>0.0122</b></td><td>0.7141</td><td>0.5249</td><td>0.0208</td><td>0.5118</td><td>0.6771</td></tr><tr><td>CNMF</td><td>3.0281</td><td>3.2180</td><td><b>0.8357</b></td><td>3.0607</td><td>5.9467</td><td>0.5370</td><td>4.3339</td><td>5.4886</td><td>0.6002</td><td><b>0.0430</b></td><td>0.7355</td><td>0.4814</td><td>0.2961</td><td>0.2906</td><td><b>0.4974</b></td></tr><tr><td>Bay-N</td><td><b>2.4809</b></td><td><b>2.5412</b></td><td><b>0.8897</b></td><td>2.5938</td><td>4.3295</td><td>0.7042</td><td>2.8266</td><td>3.3996</td><td>0.8173</td><td>0.0081</td><td>0.7255</td><td>0.5176</td><td><b>0.0073</b></td><td>0.6268</td><td>0.6042</td></tr><tr><td>Bay-S</td><td><b>2.3088</b></td><td><b>2.3741</b></td><td><b>0.8980</b></td><td>2.5185</td><td>4.2887</td><td>0.7078</td><td>2.8266</td><td>3.3989</td><td>0.8173</td><td>0.0080</td><td>0.7031</td><td>0.5383</td><td><b>0.0073</b></td><td>0.6225</td><td>0.6077</td></tr><tr><td>HySure</td><td><b>2.4888</b></td><td><b>2.7805</b></td><td><b>0.8764</b></td><td>7.4812</td><td>7.7313</td><td>0.5099</td><td>3.2765</td><td>4.3989</td><td>0.7365</td><td><b>0.0260</b></td><td>0.6875</td><td>0.5374</td><td>0.1478</td><td>0.6124</td><td>0.4897</td></tr></tbody></table><table><thead><tr><th rowspan="3"></th><th colspan="3">ALI</th><th colspan="3">WV3</th><th colspan="3">ALI</th><th colspan="3">WV3</th></tr><tr><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">MS</th><th colspan="3">PAN</th><th colspan="3">MS</th></tr><tr><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th></tr><tr><th>Optimum</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>EXP</td><td>4.0809</td><td>2.3556</td><td>0.4930</td><td>4.1208</td><td>2.3736</td><td>0.4868</td><td>4.1208</td><td>2.3736</td><td>0.4868</td><td>0.0250</td><td>0.7792</td><td>0.4524</td><td>0.0250</td><td>0.7792</td><td>0.4524</td></tr><tr><td>BT</td><td><b>4.0809</b></td><td><b>2.4750</b></td><td><b>0.6310</b></td><td>4.1208</td><td>2.6605</td><td>0.5119</td><td>4.9025</td><td>2.8810</td><td>0.5576</td><td><b>0.1255</b></td><td>0.7805</td><td>0.3831</td><td>0.2966</td><td>0.3736</td><td><b>0.4669</b></td></tr><tr><td>GSA</td><td><b>3.4450</b></td><td><b>1.8687</b></td><td><b>0.6998</b></td><td>4.0682</td><td>2.3214</td><td>0.6135</td><td>6.4658</td><td>3.6062</td><td>0.5650</td><td><b>0.0213</b></td><td>0.7350</td><td><b>0.4984</b></td><td>0.3048</td><td>0.2725</td><td>0.4944</td></tr><tr><td>SFIM</td><td>4.2951</td><td>2.1552</td><td>0.6351</td><td>4.2878</td><td>2.2921</td><td>0.6057</td><td><b>3.5968</b></td><td><b>2.0684</b></td><td><b>0.6913</b></td><td><b>0.0194</b></td><td>0.7381</td><td>0.4969</td><td>0.0236</td><td>0.5059</td><td>0.6782</td></tr><tr><td>ATWT</td><td>4.3183</td><td><b>2.1107</b></td><td><b>0.6913</b></td><td>4.5761</td><td>2.6306</td><td>0.6000</td><td><b>4.2626</b></td><td><b>2.0407</b></td><td><b>0.7212</b></td><td><b>0.0170</b></td><td>0.7408</td><td>0.4962</td><td>0.0282</td><td>0.4387</td><td>0.7177</td></tr><tr><td>GLP-HPM</td><td>4.4371</td><td>2.2155</td><td>0.6823</td><td>4.3560</td><td>2.3401</td><td>0.6214</td><td><b>3.5213</b></td><td><b>2.0407</b></td><td><b>0.7212</b></td><td><b>0.0186</b></td><td>0.7493</td><td>0.4867</td><td>0.0272</td><td>0.4599</td><td>0.7052</td></tr><tr><td>GLP-CBD</td><td><b>3.3688</b></td><td><b>1.8193</b></td><td><b>0.7261</b></td><td>3.9217</td><td>2.1277</td><td>0.6429</td><td>3.6260</td><td>2.1248</td><td>0.7234</td><td><b>0.0153</b></td><td>0.7403</td><td>0.4979</td><td>0.0246</td><td>0.4941</td><td>0.6851</td></tr><tr><td>Hyper</td><td><b>3.3684</b></td><td><b>1.8199</b></td><td><b>0.7259</b></td><td>3.9184</td><td>2.1276</td><td>0.6427</td><td>3.8306</td><td>2.2194</td><td>0.6966</td><td><b>0.0153</b></td><td>0.7403</td><td>0.4980</td><td>0.0246</td><td>0.4941</td><td>0.6851</td></tr><tr><td>CNMF</td><td>4.1532</td><td><b>2.1223</b></td><td><b>0.7023</b></td><td>4.1351</td><td>2.1607</td><td>0.5166</td><td><b>3.7804</b></td><td><b>2.6307</b></td><td>0.6027</td><td><b>0.0374</b></td><td>0.7209</td><td>0.4989</td><td>0.3091</td><td>0.2669</td><td><b>0.4917</b></td></tr><tr><td>Bay-N</td><td><b>4.2876</b></td><td><b>2.1478</b></td><td><b>0.6600</b></td><td>4.5588</td><td>2.7078</td><td>0.5355</td><td>4.3109</td><td>2.5638</td><td>0.6354</td><td><b>0.0518</b></td><td>0.7402</td><td>0.4706</td><td>0.0740</td><td>0.3873</td><td>0.6975</td></tr><tr><td>Bay-S</td><td><b>4.1775</b></td><td><b>2.1143</b></td><td><b>0.6591</b></td><td>4.5143</td><td>2.6917</td><td>0.5401</td><td>4.3115</td><td>2.5637</td><td>0.6354</td><td><b>0.0523</b></td><td>0.7293</td><td>0.4800</td><td>0.0729</td><td>0.3878</td><td>0.6985</td></tr><tr><td>HySure</td><td>5.3442</td><td>2.7851</td><td><b>0.6108</b></td><td>4.8845</td><td><b>2.7712</b></td><td>0.5303</td><td><b>4.1981</b></td><td>2.8462</td><td>0.5898</td><td><b>0.0903</b></td><td>0.7314</td><td>0.4496</td><td>0.1170</td><td>0.3774</td><td><b>0.6547</b></td></tr></tbody></table><table><thead><tr><th rowspan="3"></th><th colspan="3">ALI</th><th colspan="3">WV3</th><th colspan="3">ALI</th><th colspan="3">WV3</th></tr><tr><th colspan="3">PAN</th><th colspan="3">PAN</th><th colspan="3">MS</th><th colspan="3">PAN</th><th colspan="3">MS</th></tr><tr><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>SAM</th><th>ERGAS</th><th>Q2<sup>n</sup></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th><th>D<sub>λ</sub></th><th>D<sub>S</sub></th><th>QNR<sub>+</sub></th></tr><tr><th>Optimum</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>EXP</td><td>5.1271</td><td>1.4488</td><td>0.2097</td><td>5.1271</td><td>1.4488</td><td>0.2097</td><td>5.1271</td><td>1.4488</td><td>0.2097</td><td>0.0358</td><td>0.8182</td><td>0.4037</td><td>0.0358</td><td>0.8182</td><td>0.4037</td></tr><tr><td>BT</td><td><b>5.1271</b></td><td><b>1.4643</b></td><td><b>0.3702</b></td><td>5.1271</td><td>1.4705</td><td>0.3633</td><td>5.2432</td><td>1.5154</td><td>0.4746</td><td><b>0.1289</b></td><td>0.8130</td><td>0.3516</td><td>0.3002</td><td>0.4151</td><td><b>0.4477</b></td></tr><tr><td>GSA</td><td><b>4.7540</b></td><td><b>1.2970</b></td><td><b>0.3636</b></td><td>4.8938</td><td>1.2858</td><td>0.4259</td><td><b>4.4040</b></td><td><b>1.2643</b></td><td><b>0.5705</b></td><td><b>0.0307</b></td><td>0.7717</td><td>0.4559</td><td>0.2993</td><td>0.2770</td><td><b>0.4987</b></td></tr><tr><td>SFIM</td><td>5.4186</td><td>1.3888</td><td>0.3549</td><td>5.4085</td><td>1.3842</td><td>0.4064</td><td><b>5.1848</b></td><td><b>1.4068</b></td><td><b>0.5545</b></td><td><b>0.0299</b></td><td>0.7778</td><td>0.4545</td><td>0.0341</td><td>0.5026</td><td>0.6695</td></tr><tr><td>ATWT</td><td>5.3580</td><td>1.3772</td><td>0.3452</td><td>5.7471</td><td>1.5254</td><td>0.4484</td><td><b>4.2637</b></td><td><b>1.2216</b></td><td><b>0.5688</b></td><td><b>0.0298</b></td><td>0.7825</td><td>0.4457</td><td>0.0331</td><td>0.4791</td><td>0.6862</td></tr><tr><td>GLP-HPM</td><td>5.6302</td><td>1.4059</td><td>0.4234</td><td>5.4887</td><td>1.3907</td><td>0.4355</td><td>4.7470</td><td>1.3681</td><td><b>0.6321</b></td><td><b>0.0176</b></td><td>0.7785</td><td>0.4479</td><td>0.0305</td><td>0.4452</td><td>0.7110</td></tr><tr><td>GLP-CBD</td><td><b>4.5275</b></td><td><b>1.2246</b></td><td><b>0.4657</b></td><td>4.8538</td><td>1.3071</td><td>0.4909</td><td>4.9158</td><td>1.5127</td><td><b>0.5777</b></td><td><b>0.0664</b></td><td>0.8055</td><td>0.3978</td><td>0.1857</td><td>0.3448</td><td>0.5947</td></tr><tr><td>Hyper</td><td><b>4.5168</b></td><td><b>1.2270</b></td><td><b>0.4645</b></td><td>4.8276</td><td>1.3091</td><td>0.4890</td><td>4.5966</td><td>1.3215</td><td><b>0.6033</b></td><td><b>0.0170</b></td><td>0.7784</td><td>0.4588</td><td>0.0307</td><td>0.4657</td><td>0.6975</td></tr><tr><td>CNMF</td><td>5.0905</td><td><b>1.3107</b></td><td><b>0.4881</b></td><td>5.0738</td><td>1.4969</td><td>0.4714</td><td><b>4.7177</b></td><td>1.5112</td><td><b>0.5667</b></td><td><b>0.0418</b></td><td>0.7620</td><td><b>0.4576</b></td><td>0.3483</td><td>0.2667</td><td>0.4505</td></tr><tr><td>Bay-N</td><td>5.7306</td><td><b>1.4619</b></td><td><b>0.4276</b></td><td>5.6472</td><td>1.6148</td><td>0.4131</td><td><b>4.9158</b></td><td>1.5127</td><td><b>0.5777</b></td><td><b>0.0664</b></td><td>0.8055</td><td>0.3978</td><td>0.1857</td><td>0.3448</td><td>0.5947</td></tr><tr><td>Bay-S</td><td>5.7643</td><td><b>1.4530</b></td><td><b>0.4299</b></td><td>5.6575</td><td>1.6187</td><td>0.4151</td><td><b>4.9126</b></td><td>1.5114</td><td><b>0.5780</b></td><td><b>0.0731</b></td><td>0.7877</td><td>0.4112</td><td>0.1638</td><td>0.3485</td><td>0.6172</td></tr><tr><td>HySure</td><td>5.7783</td><td><b>1.4730</b></td><td><b>0.4586</b></td><td>5.7568</td><td>1.5847</td><td>0.4241</td><td><b>4.5480</b></td><td>1.5078</td><td><b>0.5722</b></td><td><b>0.1515</b></td><td>0.8398</td><td>0.3129</td><td>0.2243</td><td>0.3314</td><td><b>0.5586</b></td></tr></tbody></table>

Reduced Resolution

Full Resolution

In this regard, the sharpened images obtainable through the WV3 sensor, which is characterized by a two times smaller GSI with respect to the QB instrument, feature a higher visual quality.

Consistent index values are obtained also in terms of spectral distortion, since, for all the presented algorithms, a minor accuracy is achieved through the multiplatform data. Particularly evident is the case of the **GSA** algorithm, which leads to sharpened images with oversaturated colors in the multiplatform settings resulting in a final product with an anomalous balance among spectral bands.

## B. Performance Assessment With Temporally Misaligned Data

The Capodichino data set has been designed to reproduce a scenario with significant differences between the scenes observed by the multiplatform sensors. The illuminated area related to the Capodichino data set is very close to that of the Agnano data set. In particular, the HS image has been extracted by the same Hyperion acquisition, choosing a different zone with a more dynamic scenario. As it can be noticed by looking at the close-ups reported in the first row of Fig. 9 that focuses on the Naples airport, the scene

include rapidly changing objects (see the aircraft highlighted by the red rectangle) and a built area containing shapes that have been significantly modified in the period between the ALI and WV3 acquisitions (see, for example, the platform in the bottom left part of the image, highlighted by the green rectangle).

1) Reduced Resolution Assessment: The RR assessment phase allows to accurately evaluate the performance of the algorithms in a perfectly controlled scenario. Superior results are expected by employing the ALI PAN as HR source; this behavior is confirmed by the values of the quality indexes reported in Table IV. For the CS and the Bayesian algorithms, this result also extends to greater resolution enhancement factors, notwithstanding the necessity of interpolating the data to match the target scale.

2) Full Resolution Assessment: The images reported in Fig. 9 are very helpful to resume the main indications achieved in this article. In particular, the presence of the highly reflecting object in the top right part of the image is preserved by the MRA algorithms (see the results of SFIM and GLP-HPM), while it is completely removed by the **GSA** algorithm, belonging to the CS class, since it is not present in the HR images coming from the WV3 satellite. Furthermore, the aircraft platform in the bottom left is not present anymore
<figure><img src="image_9.png" /><figcaption>Fig. 6. Close-ups of the products achieved by applying the pansharpening algorithms to the FR *New York data set* (red, green, and blue bands). The resolution enhancement factor is *R* = 6. (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). (a) EXP. (b) ALI PAN. (c) WV3 PAN. (d) WV3 MS. (e) GSA ALI PAN. (f) SFIM ALI PAN. (g) GLP-HPM ALI PAN. h) Hyper ALI PAN. (i) GSA WV3 PAN. (j) SFIM WV3 PAN. (k) GLP-HPM WV3 PAN. (l) Hyper WV3 PAN. (m) GSA WV3 MS. (n) SFIM WV3 MS. (o) GLP-HPM WV3 MS. (p) Hyper WV3 MS.</figcaption></figure>

in the images acquired in 2015, but it is reintroduced by
the GSA fusion algorithm. The presented scenario includes
intentionally exaggerated temporal variations. However, mul-
tiplatform acquisitions are not always simultaneous and, this
proves that the MRA algorithms are more advisable in this
particular case.

In fact, the images illustrate as the CS methods are very
prone to introduce an excessive quantity of details, which
render the image sharper but often oversaturated. The effect of
this issue is acceptable only in a single platform environment,
instead reducing the quality of the final products at resolution

enhancement factors *R* greater than 3 (as in the case of
Fig. 9). Furthermore, Fig. 9 allows us to gain more insight
from multiplatform data fusion illustrating the importance
of employing an MS image. The comparison of the images
depicted in the fourth row, corresponding to HS/MS fusion,
to the images reported in the third row, corresponding to
HS/PAN fusion, points out the more natural representation
of the shapes and colors achievable by employing the more
specific MS information, which allows to avoid the cartoonized
effect present in the images that are produced using the PAN
image as HR source.
<figure><img src="image_10.png" /><figcaption>Fig. 7. Close-ups of the products achieved by applying the pansharpening algorithms to the FR *Agnano data set* (red, green, and blue bands). The resolution enhancement factor is *R* = 6. (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). (a) EXP. (b) ALI PAN. (c) WV3 PAN. (d) WV3 MS. (e) GSA ALI PAN. (f) SFIM ALI PAN. (g) GLP-HPM ALI PAN. (h) Hyper ALI PAN. (i) GSA WV3 PAN. (j) SFIM WV3 PAN. (k) GLP-HPM WV3 PAN. (l) Hyper WV3 PAN. (m) GSA WV3 MS. (n) SFIM WV3 MS. (o) GLP-HPM WV3 MS. (p) Hyper WV3 MS.</figcaption></figure>

### C. Concluding Remarks

The last analysis is about the algorithms' performance. It has been performed in two separate frameworks. In fact, on the one hand, it is necessary to assess the performance of the sharpening algorithms in the most typical setting, namely by employing data sets composed by images that are acquired on the same area of the Earth surface in slightly different time instants, so that the objects in the acquired scenes are very similar to each other. On the other hand, it is very interesting to illustrate the results obtainable in a purposely designed test that is based on a group of images characterized

by a significant change between the scenes acquired by the two sensors mounted on board of the two different satellite platforms.

The first analysis evidences that the use of multichannel multiplatform data is almost always advisable. This consideration can be justified by considering the higher specificity of the spatial information carried by the MS images, which are acquired in a frequency range more similar to that of the enhanced HS channel. This result is common to all the three employed data sets, also for the resolution enhancement factors that do not require the upsampling of
<figure><img src="image_11.png" /><figcaption>Fig. 8. Close-ups of the products achieved by applying the pansharpening algorithms to the FR *San Francisco data set* (red, green, and blue bands). The resolution enhancement factor is *R* = 6. (a) EXP. (b) ALI PAN. (c) QB PAN. (d) QB MS. (e) GSA ALI PAN. (f) SFIM ALI PAN. (g) GLP-HPM ALI PAN. (h) Hyper ALI PAN. (i) GSA QB PAN. (j) SFIM QB PAN. (k) GLP-HPM QB PAN. (l) Hyper WV3 PAN. (m) GSA QB MS. (n) SFIM QB MS. (o) GLP-HPM QB MS. (p) Hyper WV3 MS.</figcaption></figure>

the high-resolution image acquired by the same platform (i.e., for *R* = 3). Such behavior is particularly evident for the MRA methods, which obtain always the best results employing MS multiplatform data. This is true both for lower performance MRA algorithms (e.g., SFIM) and higher performance MRA methods (e.g., GLP-HPM), as clearly corroborated by the qualitative analysis of Figs. 6–8. Instead, for the CS and Bayesian methods, this outcome is more evident at FR, where a higher presence of spatial details is experienced.

The second analysis highlights a different behavior when temporal misalignments occur. In that case, the use of the images acquired by the same platform is often advisable,

to avoid the introduction of artifacts produced by objects only visible in the HR images. In such situations, the use of CS methods is deprecated, since they strongly modify both the spatial and the spectral content of the HS image forcing the introduction of features only present in the high resolution (temporally misaligned) data. On the contrary, the application of MRA and Bayesian method in conjunction with multiplatform data is much more debatable. The comparison between the RR and FR assessment results indicate that their use with data acquired by a different platform becomes more valuable for finer spatial resolutions to be reached, namely with the increase of the spatial information that cannot be provided by the ALI PAN image.
<figure><img src="image_12.png" /><figcaption>Fig. 9. Close-ups of the products achieved by applying the pansharpening algorithms to the FR *Capodichino data set* (red, green, and blue bands). The resolution enhancement factor is *R* = 6. (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The red and the green rectangles highlight two areas of the scene that are significantly different across the EO-1 and WV3 acquisitions. (a) EXP. (b) ALI PAN. (c) WV3 PAN. (d) WV3 MS. (e) GSA ALI PAN. (f) SFIM ALI PAN. (g) GLP-HPM ALI PAN. (h) Hyper ALI PAN. (i) GSA WV3 PAN. (j) SFIM WV3 PAN. (k) GLP-HPM WV3 PAN. (l) Hyper WV3 PAN. (m) GSA WV3 MS. (n) SFIM WV3 MS. (o) GLP-HPM WV3 MS. (p) Hyper WV3 MS.</figcaption></figure>

VI. CONCLUSION

The goal of this article was to present the first deep study about the sharpening of satellite HS images in practical settings. Accordingly, the key aspect of the presented study was the utilization of real data, acquired by the EO-1, the Worldview-3, and the Quickbird sensors. The benefits of using images acquired by a different platform for enhancing the Hyperion data have been highlighted in the perspective of reaching high-resolution enhancement factors and of introducing more proper spatial information content. Several

operational difficulties have been handled in this article. The most important ones are the necessity of dealing with data acquired in different times and the complexity of individuating a HR reference for assessing the spatial quality of the fused products at FR.

REFERENCES

[1] G. Camps-Valls, D. Tuia, L. Bruzzone, and J. A. Benediktsson, "Advances in hyperspectral image classification: Earth monitoring with statistical learning methods," *IEEE Signal Process. Mag.*, vol. 31, no. 1, pp. 45-54, Jan. 2014.
[2] J. Transon, R. d'Andrimont, A. Maugnard, and P. Defourny, "Survey of hyperspectral earth observation applications from space in the Sentinel-2 context," *Remote Sens.*, vol. 10, no. 3, p. 157, Jan. 2018.

[3] I. Mariotto, P. S. Thenkabail, A. Huete, E. T. Slonecker, and A. Platonov, "Hyperspectral versus multispectral crop-productivity modeling and type discrimination for the HyspIRI mission," *Remote Sens. Environ.*, vol. 139, pp. 291-305, Dec. 2013.

[4] M. Marshall and P. Thenkabail, "Advantage of hyperspectral EO-1 hyperion over multispectral IKONOS, GeoEye-1, WorldView-2, Landsat ETM+, and MODIS vegetation indices in crop biomass estimation," *ISPRS J. Photogramm. Remote Sens.*, vol. 108, pp. 205-218, Oct. 2015.

[5] C. Gomez, R. A. Viscarra Rossel, and A. B. McBratney, "Soil organic carbon prediction by hyperspectral remote sensing and field vis-NIR spectroscopy: An Australian case study," *Geoderma*, vol. 146, nos. 3-4, pp. 403-411, Aug. 2008.

[6] G. Carter et al., "Remote sensing and mapping of tamarisk along the colorado river, USA: A comparative use of summer-acquired hyperion, thematic mapper and quickbird data," *Remote Sens.*, vol. 1, no. 3, pp. 318-329, Jul. 2009.

[7] A. Okujeni, S. van der Linden, and P. Hostert, "Extending the vegetation-impervious-soil model using simulated EnMAP data and machine learning," *Remote Sens. Environ.*, vol. 158, pp. 69-80, Mar. 2015.

[8] B. Siegmann, T. Jarmer, F. Beyer, and M. Ehlers, "The potential of pan-sharpened EnMAP data for the assessment of wheat LAI," *Remote Sens.*, vol. 7, no. 10, pp. 12737-12762, Oct. 2015.

[9] N. Yokoya, J. Chan, and K. Segl, "Potential of resolution-enhanced hyperspectral data for mineral mapping using simulated EnMAP and Sentinel-2 images," *Remote Sens.*, vol. 8, no. 3, p. 172, Feb. 2016.

[10] S. J. Walsh et al., "QuickBird and hyperion data analysis of an invasive plant species in the Galapagos islands of equador: Implications for control and land use management," *Remote Sens. Environ.*, vol. 112, no. 5, pp. 1927-1941, May 2008.

[11] L. Loncan et al., "Hyperspectral pansharpening: A review," *IEEE Trans. Geosci. Remote Sens.*, vol. 3, no. 3, pp. 27-46, Sep. 2015.

[12] N. Yokoya, C. Grohnfeldt, and J. Chanussot, "Hyperspectral and multispectral data fusion: A comparative review of the recent literature," *IEEE Geosci. Remote Sens. Mag.*, vol. 5, no. 2, pp. 29-56, Jun. 2017.

[13] Y. Zhang, S. De Backer, and P. Scheunders, "Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images," *IEEE Trans. Geosci. Remote Sens.*, vol. 47, no. 11, pp. 3834-3843, Nov. 2009.

[14] D. Picone, R. Restaino, G. Vivone, P. Addesso, and J. Chanussot, "Pansharpening of hyperspectral images: Exploiting data acquired by multiple platforms," in *Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS)*, Jul. 2016, pp. 7220-7223.

[15] J. Yang, Y.-Q. Zhao, and J. Chan, "Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network," *Remote Sens.*, vol. 10, no. 5, p. 800, May 2018.

[16] J. Marcello, E. Ibarrola-Ulzurrun, C. Gonzalo-Martin, J. Chanussot, and G. Vivone, "Assessment of hyperspectral sharpening methods for the monitoring of natural areas using multiplatform remote sensing imagery," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 10, pp. 8208-8222, Oct. 2019.

[17] G. Vivone, R. Restaino, G. Licciardi, M. Dalla Mura, and J. Chanussot, "MultiResolution analysis and component substitution techniques for hyperspectral pansharpening," in *Proc. IEEE Geosci. Remote Sens. Symp.*, Jul. 2014, pp. 2649-2652.

[18] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, "Twenty-five years of pansharpening: A critical review and new developments," in *Signal and Image Processing for Remote Sensing*, 2nd ed., C.-H. Chen, Ed. Boca Raton, FL, USA: CRC Press, 2012, pp. 533-548.

[19] G. Vivone, R. Restaino, M. Dalla Mura, G. Licciardi, and J. Chanussot, "Contrast and error-based fusion schemes for multispectral image pansharpening," *IEEE Geosci. Remote Sens. Lett.*, vol. 11, no. 5, pp. 930-934, May 2014.

[20] M. M. Khan, J. Chanussot, and L. Alparone, "Pansharpening of hyperspectral images using spatial distortion optimization," in *Proc. 16th IEEE Int. Conf. Image Process. (ICIP)*, Nov. 2009, pp. 2853-2858.

[21] G. A. Licciardi, M. M. Khan, and J. Chanussot, "Fusion of hyperspectral and panchromatic images: A hybrid use of inclusion and nonlinear PCA," in *Proc. 19th IEEE Int. Conf. Image Process.*, Sep. 2012, pp. 2133-2136.

[22] B. Aiazzi, L. Alparone, S. Baronti, L. Santurri, and M. Selva, "Spatial resolution enhancement of ASTER thermal bands," *Proc. SPIE*, vol. 5982, pp. 59821G-1-59821G-10, Oct. 2005.

[23] D. Picone, R. Restaino, G. Vivone, P. Addesso, M. D. Mura, and J. Chanussot, "Band assignment approaches for hyperspectral sharpening," *IEEE Geosci. Remote Sens. Lett.*, vol. 14, no. 5, pp. 739-743, May 2017.

[24] M. Selva, B. Aiazzi, F. Butera, L. Chiarantini, and S. Baronti, "Hyper-sharpening: A first approach on SIM-GA data," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 8, no. 6, pp. 3008-3024, Jun. 2015.

[25] M. Selva, L. Santurri, and S. Baronti, "Improving hypersharpening for WorldView-3 data," *IEEE Geosci. Remote Sens. Lett.*, vol. 16, no. 6, pp. 987-991, Jun. 2019.

[26] R. C. Hardie, M. T. Eismann, and G. L. Wilson, "MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor," *IEEE Trans. Image Process.*, vol. 13, no. 9, pp. 1174-1184, Sep. 2004.

[27] Q. Wei, J. Bioucas-Dias, N. Dobigeon, and J.-Y. Tourneret, "Hyperspectral and multispectral image fusion based on a sparse representation," *IEEE Trans. Geosci. Remote Sens.*, vol. 53, no. 7, pp. 3658-3668, Jul. 2015.

[28] M. Simoes, J. Bioucas-Dias, L. B. Almeida, and J. Chanussot, "A convex formulation for hyperspectral image superresolution via subspace-based regularization," *IEEE Trans. Geosci. Remote Sens.*, vol. 53, no. 6, pp. 3373-3388, Jun. 2015.

[29] M. T. Eismann and R. C. Hardie, "Application of the stochastic mixing model to hyperspectral resolution enhancement," *IEEE Trans. Geosci. Remote Sens.*, vol. 42, no. 9, pp. 1924-1933, Sep. 2004.

[30] N. Yokoya, T. Yairi, and A. Iwasaki, "Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion," *IEEE Trans. Geosci. Remote Sens.*, vol. 50, no. 2, pp. 528-537, Feb. 2012.

[31] G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa, "Pansharpening by convolutional neural networks," *Remote Sens.*, vol. 8, no. 7, p. 594, Jul. 2016.

[32] F. Palsson, J. R. Sveinsson, M. O. Ulfarsson, and J. A. Benediktsson, "Model-based fusion of multi- and hyperspectral images using PCA and wavelets," *IEEE Trans. Geosci. Remote Sens.*, vol. 53, no. 5, pp. 2652-2663, May 2015.

[33] R. Dian, S. Li, A. Guo, and L. Fang, "Deep hyperspectral image sharpening," *IEEE Trans. Neural Netw. Learn. Syst.*, vol. 29, no. 11, pp. 5345-5355, Nov. 2018.

[34] L. He, J. Zhu, J. Li, A. Plaza, J. Chanussot, and B. Li, "A theoretical analysis of the effects of aliasing and misregistration on pansharpened imagery," *IEEE J. Sel. Topics Signal Process.*, vol. 12, no. 8, pp. 3092-3100, Aug. 2019.

[35] G. Vivone et al., "A critical comparison among pansharpening algorithms," *IEEE Trans. Geosci. Remote Sens.*, vol. 53, no. 5, pp. 2565-2586, May 2015.

[36] L. Wald, T. Ranchin, and M. Mangolini, "Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images," *Photogramm. Eng. Remote Sens.*, vol. 63, no. 6, pp. 691-699, 1997.

[37] M. M. Khan, L. Alparone, and J. Chanussot, "Pansharpening quality assessment using the modulation transfer functions of instruments," *IEEE Trans. Geosci. Remote Sens.*, vol. 47, no. 11, pp. 3880-3891, Nov. 2009.

[38] C. Thomas, T. Ranchin, L. Wald, and J. Chanussot, "Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics," *IEEE Trans. Geosci. Remote Sens.*, vol. 46, no. 5, pp. 1301-1312, May 2008.

[39] R. M. Dudley, *Real Analysis and Probability*. Cambridge, U.K.: Cambridge Univ. Press, 2004.

[40] A. L. Gibbs and F. E. Su, "On choosing and bounding probability metrics," *Int. Stat. Rev.*, vol. 70, no. 3, pp. 419-435, Dec. 2002.

[41] R. L. Dobrushin, "Prescribing a system of random variables by conditional distributions," *Theory Probab. Appl.*, vol. 15, no. 3, pp. 458-486, Jan. 1970.

[42] S. Kolouri, S. R. Park, M. Thorpe, D. Slepcev, and G. K. Rohde, "Optimal mass transport: Signal processing and machine-learning applications," *IEEE Signal Process. Mag.*, vol. 34, no. 4, pp. 43-59, Jul. 2017.

[43] S. N. Evans and F. A. Matsen, "The phylogenetic Kantorovich-Rubinstein metric for environmental sequence samples," *J. Roy. Stat. Soc. B, Stat. Methodol.*, vol. 74, no. 3, pp. 569-592, Jun. 2012.

[44] M. Sommerfeld and A. Munk, "Inference for empirical wasserstein distances on finite spaces," *J. Roy. Stat. Soc. B, Stat. Methodol.*, vol. 80, no. 1, pp. 219-238, Jan. 2018.

[45] C. Villani, *Optimal Transport: Old and New*. Berlin, Germany: Springer, 2009.
[46] S. Peleg, M. Werman, and H. Rom, "A unified approach to the change of resolution: Space and gray-level," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 11, no. 7, pp. 739-742, Jul. 1989.

[47] S. Haker, L. Zhu, A. Tannenbaum, and S. Angenent, "Optimal mass transport for registration and warping," *Int. J. Comput. Vis.*, vol. 60, no. 3, pp. 225-240, Dec. 2004.

[48] Y. Rubner, C. Tomasi, and L. J. Guibas, "The earth mover's distance as a metric for image retrieval," *Int. J. Comput. Vis.*, vol. 40, no. 2, pp. 99-121, Nov. 2000.

[49] J. Delon, "Midway image equalization," *J. Math. Imag. Vis.*, vol. 21, no. 2, pp. 119-134, Sep. 2004.

[50] J. Rabin, S. Ferradans, and N. Papadakis, "Adaptive color transfer with relaxed optimal transport," in *Proc. IEEE Int. Conf. Image Process. (ICIP)*, Oct. 2014, pp. 4852-4856.

[51] K. Ni, X. Bresson, T. Chan, and S. Esedoglu, "Local histogram based segmentation using the Wasserstein distance," *Int. J. Comput. Vis.*, vol. 84, no. 1, pp. 97-111, Aug. 2009.

[52] W. J. Carper, T. M. Lillesand, and R. W. Kiefer, "The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data," *Photogramm. Eng. Remote Sens.*, vol. 56, no. 4, pp. 459-467, Apr. 1990.

[53] P. Chavez, Jr., S. C. Sides, and J. A. Anderson, "Comparison of three different methods to merge multiresolution and multispectral data-Landsat TM and SPOT panchromatic," *Photogramm. Eng. Remote Sens.*, vol. 57, no. 3, pp. 295-303, Mar. 1991.

[54] P. S. Chavez, Jr., and A. Y. Kwarteng, "Extracting spectral contrast in Landsat thematic map image data using selective principal component analysis," *Photogramm. Eng. Remote Sens.*, vol. 55, no. 3, pp. 339-348, 1989.

[55] V. K. Shettigara, "A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set," *Photogram. Eng. Remote Sens.*, vol. 58, no. 5, pp. 561-567, 1992.

[56] V. P. Shah, N. H. Younan, and R. L. King, "An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets," *IEEE Trans. Geosci. Remote Sens.*, vol. 46, no. 5, pp. 1323-1335, May 2008.

[57] C. A. Laben and B. V. Brower, "Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening," U.S. Patent 6011875 A, Jan. 4, 2000.

[58] J. G. Liu, "Smoothing filter based intensity modulation: A spectral preserve image fusion technique for improving spatial details," *Int. J. Remote Sens.*, vol. 21, no. 18, pp. 3461-3472, Dec. 2000.

[59] L. Wald and T. Ranchin, "Liu 'Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details,'" *Int. J. Remote Sens.*, vol. 23, no. 3, pp. 593-597, Jan. 2002.

[60] P. Burt and E. Adelson, "The Laplacian pyramid as a compact image code," *IEEE Trans. Commun.*, vol. COM-31, no. 4, pp. 532-540, Apr. 1983.

[61] X. Otazu, M. Gonzalez-Audicana, O. Fors, and J. Nunez, "Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods," *IEEE Trans. Geosci. Remote Sens.*, vol. 43, no. 10, pp. 2376-2385, Oct. 2005.

[62] S. Baronti, B. Aiazzi, M. Selva, A. Garzelli, and L. Alparone, "A theoretical analysis of the effects of aliasing and misregistration on pansharpened imagery," *IEEE J. Sel. Topics Signal Process.*, vol. 5, no. 3, pp. 446-453, Jun. 2011.

[63] B. Aiazzi, L. Alparone, S. Baronti, R. Carla, A. Garzelli, and L. Santurri, "Sensitivity of pansharpening methods to temporal and instrumental changes between multispectral and panchromatic data sets," *IEEE Trans. Geosci. Remote Sens.*, vol. 55, no. 1, pp. 308-319, Jan. 2017.

[64] R. A. Schowengerdt, *Remote Sensing: Models and Methods for Image Processing*, 3rd ed. Orlando, FL, USA: Elsevier, 2007.

[65] A. K. Jain, *Fundamentals of Digital Image Processing*. Englewood Cliffs, NJ, USA: Prentice-Hall, 1989.

[66] J. M. Boucas-Dias et al., "Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 5, no. 2, pp. 354-379, Apr. 2012.

[67] Q. Wei, N. Dobigeon, and J.-Y. Tourneret, "Fast fusion of multi-band images based on solving a Sylvester equation," *IEEE Trans. Image Process.*, vol. 24, no. 11, pp. 4109-4121, Nov. 2015.

[68] Q. Wei, N. Dobigeon, and J. Tourneret, "Bayesian fusion of multi-band images," *IEEE J. Sel. Topics Signal Process.*, vol. 9, no. 6, pp. 1117-1127, Sep. 2015.

[69] L. Alparone, B. Aiazzi, S. Baronti, A. Garzelli, F. Nencini, and M. Selva, "Multispectral and panchromatic data fusion assessment without reference," *Photogramm. Eng. Remote Sens.*, vol. 74, no. 2, pp. 193-200, Feb. 2008.

[70] B. Aiazzi, L. Alparone, S. Baronti, R. Carlà, A. Garzelli, and L. Santurri, "Full-scale assessment of pansharpening methods and data products," *Proc. SPIE*, vol. 9244, pp. 924402-1-924402-22, Oct. 2014.

[71] L. Alparone, A. Garzelli, and G. Vivone, "Spatial consistency for full-scale assessment of pansharpening," in *Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS)*, Jul. 2018, pp. 5132-5134.

[72] G. Dall'Aglio, "Sugli extremi dei momenti delle funzioni di ripartizione doppia," *Ann. Scuola Normale Superiore Pisa*, vol. 10, nos. 1-2, pp. 35-74, Mar. 1956.

[73] C. R. Givens and R. M. Shortt, "A class of wasserstein metrics for probability distributions," *Michigan Math. J.*, vol. 31, no. 2, pp. 231-240, 1984.

[74] R. H. Yuhas, A. F. Goetz, and J. W. Boardman, "Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm," in *Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop*, 1992, pp. 147-149.

[75] L. Wald, *Data Fusion Definitions and Architectures-Fusion of Images of Different Spatial Resolutions*. Paris, France: Les Presses de l'École des Mines, 2002.

[76] A. Garzelli and F. Nencini, "Hypercomplex quality assessment of multi/hyperspectral images," *IEEE Geosci. Remote Sens. Lett.*, vol. 6, no. 4, pp. 662-665, Oct. 2009.

[77] A. R. Gillespie, A. B. Kahle, and R. E. Walker, "Color enhancement of highly correlated images. II. Channel ratio and 'chromaticity' transformation techniques," *Remote Sens. Environ.*, vol. 22, no. 3, pp. 343-365, Aug. 1987.

[78] B. Aiazzi, S. Baronti, and M. Selva, "Improving component substitution pansharpening through multivariate regression of MS +Pan data," *IEEE Trans. Geosci. Remote Sens.*, vol. 45, no. 10, pp. 3230-3239, Oct. 2007.

[79] B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, "Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis," *IEEE Trans. Geosci. Remote Sens.*, vol. 40, no. 10, pp. 2300-2312, Jan. 2002.

[80] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, "MTF-tailored multiscale fusion of high-resolution MS and pan imagery," *Photogramm. Eng. Remote Sens.*, vol. 72, no. 5, pp. 591-596, May 2006.

[81] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, "An MTF-based spectral distortion minimizing model for pansharpening of very high resolution multispectral images of urban areas," in *Proc. 22nd Digit. Avionics Syst. Conf.*, 2003, pp. 90-94.

[82] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce, "Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest," *IEEE Trans. Geosci. Remote Sens.*, vol. 45, no. 10, pp. 3012-3021, Oct. 2007.

[83] S. Lolli, L. Alparone, A. Garzelli, and G. Vivone, "Haze correction for contrast-based multispectral pansharpening," *IEEE Geosci. Remote Sens. Lett.*, vol. 14, no. 12, pp. 2255-2259, Dec. 2017.

[84] A. Garzelli, B. Aiazzi, L. Alparone, S. Lolli, and G. Vivone, "Multispectral pansharpening with radiative transfer-based detail-injection modeling for preserving changes in vegetation cover," *Remote Sens.*, vol. 10, no. 8, p. 1308, Aug. 2018.

<figure><img src="image_13.png" /></figure>

**Rocco Restaino** (Senior Member, IEEE) received the Laurea degree in electronic engineering from the University of Naples, Naples, Italy, in 1998, and the Ph.D. degree in information engineering from the University of Salerno, Fisciano, Italy, in 2002.

He is an Assistant Professor with the University of Salerno. His research interests include probability theory, stochastic geometry and signal processing for remote sensing and networking.

Dr. Restaino was a recipient of the IGARSS 2015 Symposium Best Paper Award.
<figure><img src="image_14.png" /></figure>

**Gemine Vivone** (Senior Member, IEEE) received the B.Sc. and M.Sc. degrees (*summa cum laude*) and the Ph.D. degree (Hons.) in information engineering from the University of Salerno, Fisciano, Italy, in 2008, 2011, and 2014, respectively.

In 2012, he was a Visiting Researcher with the NATO Undersea Research Center, La Spezia, Italy. In 2013, he was a Visiting Scholar with the Grenoble Institute of Technology (INPG), Grenoble, France. In 2014, he joined the North Atlantic Treaty Organization (NATO) Science & Technology Organization (STO) Center for Maritime Research and Experimentation (CMRE), La Spezia, as a Scientist. In 2019, he was an Assistant Professor with the University of Salerno. He is a Researcher with the National Research Council (Italy) and an Adjunct Professor with the University of Salerno. His main research interests focus on statistical signal processing, detection of remotely sensed images, data fusion, and tracking algorithms.

Dr. Vivone is the Leader of the Image and Signal Processing Working Group of the IEEE Image Analysis and Data Fusion Technical Committee. He is an Editorial Board Member for MDPI *Remote Sensing* and an Advisory Board Member for MDPI *Encyclopedia*. He received the Symposium Best Paper Award at IEEE International Geoscience and Remote Sensing Symposium (IGARSS) in 2015 and the Best Reviewer Award of the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING in 2017. He is an Associate Editor of IEEE GEOSCIENCE AND REMOTE SENSING LETTERS (GRSL). He served as a Guest Associate Editor for several Special Issues.

<figure><img src="image_15.png" /></figure>

**Paolo Addesso** (Member, IEEE) received the Laurea degree (*cum laude*) in electronic engineering and the Ph.D. degree in information engineering from the University of Salerno, Fisciano, Italy, in 2000 and 2005, respectively.

He is an Assistant Professor with the University of Salerno. His research interests include remote sensing, image and signal processing.

<figure><img src="image_16.png" /></figure>

**Jocelyn Chanussot** (Fellow, IEEE) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology (Grenoble INP), Grenoble, France, in 1995, and the Ph.D. degree from the Université de Savoie, Annecy, France, in 1998.

Since 1999, he has been with Grenoble INP, where he is a Professor of signal and image processing. He has been a Visiting Scholar with Stanford University, Stanford, CA, USA, KTH, Stockholm, Sweden, and NUS, Singapore. Since 2013, he has been an Adjunct Professor with the University of Iceland, Reykjavik, Iceland. From 2015 to 2017, he was a Visiting Professor with the University of California at Los Angeles (UCLA), Los Angeles, CA, USA. He holds the AXA Chair in *Remote Sensing* and an Adjunct Professor with the Chinese Academy of Sciences, Aerospace Information Research Institute, Beijing, China. His research interests include image analysis, hyperspectral remote sensing, data fusion, machine learning and artificial intelligence.

Dr. Chanussot is a member of the Institut Universitaire de France from 2012 to 2017 and a Highly Cited Researcher (Clarivate Analytics/Thomson Reuters from 2018 to 2019). He was a member of the Machine Learning for Signal Processing Technical Committee of the IEEE Signal Processing Society from 2006 to 2008 and the Program Chair of the IEEE International Workshop on Machine Learning for Signal Processing in 2009. He is the Founding President of the IEEE Geoscience and Remote Sensing French Chapter from 2007 to 2010, which received the 2010 IEEE GRS-S Chapter Excellence Award. He has received multiple outstanding paper awards. He was the Vice-President of the IEEE Geoscience and Remote Sensing Society (GRSS), in charge of meetings and symposia from 2017 to 2019. He was the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution in Remote sensing (WHISPERS). He was the Co-Chair from 2005 to 2008 and the Chair of the GRS Data Fusion Technical Committee from 2009 to 2011. He is an Associate Editor of the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, the IEEE TRANSACTIONS ON IMAGE PROCESSING, and the PROCEEDINGS OF THE IEEE. He was the Editor-in-Chief of the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING from 2011 to 2015. In 2014, he served as a Guest Editor for the *IEEE Signal Processing Magazine*.