# Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution With Subpixel Fusion

Danfeng Hong, Jing Yao, Chenyu Li, Deyu Meng, Naoto Yokoya, Jocelyn Chanussot

<figure><img src="image_1.png" /></figure>

To cite this version:

Danfeng Hong, Jing Yao, Chenyu Li, Deyu Meng, Naoto Yokoya, et al., Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution With Subpixel Fusion. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61, pp.1-12. 10.1109/TGRS.2023.3324497. hal-04473735

HAL Id: hal-04473735

https://hal.science/hal-04473735v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution with Subpixel Fusion

Danfeng Hong, *Senior Member, IEEE*, Jing Yao, Deyu Meng, *Member, IEEE*, Naoto Yokoya, *Member, IEEE*, and Jocelyn Chanussot, *Fellow, IEEE*

**Abstract**—Enormous efforts have been recently made to super-resolve hyperspectral (HS) images with the aid of high spatial resolution multispectral (MS) images. Most prior works usually perform the fusion task by means of multifarious pixel-level priors. Yet the intrinsic effects of a large distribution gap between HS-MS data due to differences in the spatial and spectral resolution are less investigated. The gap might be caused by unknown sensor-specific properties or highly-mixed spectral information within one pixel (due to low spatial resolution). To this end, we propose a subpixel-level HS super-resolution framework by devising a novel decoupled-and-coupled network, called DC-Net, to progressively fuse HS-MS information from the pixel- to subpixel-level, from the image- to feature-level. As the name suggests, DC-Net first decouples the input into common (or cross-sensor) and sensor-specific components to eliminate the gap between HS-MS images before further fusion, and then fully blends them by a model-guided coupled spectral unmixing (CSU) net. More significantly, we append a self-supervised learning module behind the CSU net by guaranteeing the material consistency to enhance the detailed appearances of the restored HS product. Extensive experimental results show the superiority of our method both visually and quantitatively and achieve a significant improvement in comparison with the state-of-the-arts. Furthermore, the codes and datasets will be available at https://sites.google.com/view/danfeng-hong for the sake of reproducibility.

**Index Terms**—Data fusion, deep Learning, hyperspectral image, self-supervised, spectral unmixing, super-resolution.

## I. INTRODUCTION

CENT Hyperspectral (HS) images are characterized by abundant and detailed spectral information, which enables the recognition of materials at a more accurate level compared to RGB or multispectral (MS) images. In recent years, HS imaging has been garnering increasing attention in a wide range of applications related to computer vision, such

This work was supported by the MIAI @ Grenoble Alpes (ANR-19-P3IA-0003), and the AXA Research.

D. Hong and J. Yao are with the Key Laboratory of Computational Optical Imaging Technology, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China (e-mail: hongdf@aircas.ac.cn; yaojing@aircas.ac.cn).

D. Meng is with the School of Mathematics and Statistics, Xi'an Jiaotong University, 710049 Xi'an, China. (e-mail: dymeng@mail.xjtu.edu.cn)

N. Yokoya is with Graduate School of Frontier Sciences, the University of Tokyo, 277-8561 Chiba, Japan, and also with the Geoinformatics Unit, RIKEN Center for Advanced Intelligence Project (AIP), RIKEN, 103-0027 Tokyo, Japan. (e-mail: naoto.yokoya@riken.jp)

J. Chanussot is with the Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, 38000 Grenoble, France (e-mail: jocelyn@hi.is).

<figure><img src="image_3.png" /><figcaption>Fig. 1. (a) shows the observation models for HrMS and LrHS images and illustrates potential challenges (highly mixed spectral signatures) in HS-SR. (b) provides a solution by coupled spectral unmixing (CSU) to decompose mixed pixels to pure and informative subpixels, yielding a subpixel-level HS-SR approach.</figcaption></figure>

as image restoration [1]–[3], image super-resolution [4]–[8], object detection and tracking [9]–[13], to name a few.

HS images are capable of capturing more subtle discrepancies between different objects, which, to a great extent, is benefited from its high spectral resolution. As a trade-off, HS imaging systems are usually designed to acquire the data at a high spatial sampling distance (i.e., sacrificing spatial resolution), thereby limiting the range of potential applications in practice [14]. But fortunately, MS imaging systems, with the broader spectral bandwidth, can provide finer spatial information. Therefore, fusing the low-resolution HS (LrHS) and high-resolution MS (HrMS) image pair is an intuitive and feasible solution to generate the high-resolution HS (HrHS) product, also known as MS-guided HS super-resolution (HS-SR). Fig. 1 (a) illustrates the observation models for HrMS and LrHS images degraded from the HrHS image.

Over the last decade, a variety of model-driven HS-SR algorithms have been successfully developed [15]. These ap-
proaches model the underlying relationships between HS-MS data by various hand-crafted priors and the knowledge of relevant sensor characteristics, e.g., spectral response function (SRF) and point spread function (PSF) of different camera systems. Very recently, the great success of convolutional neural networks (CNNs) has also made significant progress on the HS-SR task [16]–[21]. Owing to the powerful representation ability, CNNs-based models can excavate the intrinsic properties that lie in the HS (or MS) image more effectively and blend HS and MS images in a more compact way. Nevertheless, those existing, either model-based or data-driven (e.g., CNNs), fusion approaches rarely consider underlying problems that impact the HS-RS performance to a great extent, e.g., different sensor-specific properties leading to the weak affinity between original HS and MS data, highly-mixed pixels widely existed in both HS and MS images (see Fig. 1 (a)).

To this end, we propose a decoupled-and-coupled network (DC-Net) for the HS-SR task. DC-Net is a novel progressive fusion framework from the pixel- to subpixel-level fusion, from the image- to feature-level fusion. As the name suggests, we first design a decoupled subnetwork (D-Net) by means of generative adversarial networks (GANs) [22]. The D-Net disentangles the input HS and MS images to common (or cross-sensor) and sensor-specific components and recombines them in a new image space, where the sensor-specific information is preliminary exchanged to enable a more tight connection of HS-MS information, further providing a better starting point for the subsequent fusion process. Inspired by spectral unmixing [23], we then develop a model-driven coupled subnetwork (C-Net) on the output of the D-Net to address the mixed pixel problem for a more compact fusion of HS-MS images. Fig. 1 (b) illustrates a subpixel-level solution for HS-SR by the CSU net. More significantly, a self-supervised learning module (S-Net) is deployed to the C-Net to further enhance detailed geometries and textures, yielding more realistic HrHS images. The main contributions of this paper are three-fold as follows.

* Beyond the pixel-level fusion, we provide a new subpixel insight for the HS-SR task by developing a unified decoupled-and-coupled network architecture in an end-to-end fashion, DC-Net for short.

* DC-Net is a progressive fusion framework, which not only can eliminate the gap between HS-MS images for better pixel-level information exchange (from the image perspective via D-Net) but also can alleviate the effects of highly mixed spectral pixels (due to low spatial resolution) for a more compact subpixel-level fusion (from the feature perspective via C-Net).

* A plug-and-play self-supervised module, i.e., S-Net, is devised in C-Net by aligning the abundance maps obtained by unmixing HS and MS images, respectively, to maintain semantic consistency between abundance maps of HS-MS images, further improving the quality of the HrHS product.

The rest of this paper is organized as follows. Section 2 introduces the related works from the perspectives of traditional models and deep models in detail. We then elaborate the proposed DC-Net and details each newly-developed mod-

ules. Experiential results are conducted in comparison with currently existing state-of-the-art HS-SR approaches in Section 4. Finally, Section 5 draws a conclusion with a possible future outlook.

## II. RELATED WORK

### A. Traditional Models

HS pansharpening is a heuristic way to perform the HS-MS fusion [24], which has been widely applied in the HS image super-resolution task. Component substitution (CS) and multiresolution analysis (MRA) are the two main types of pansharpening techniques. The former one aims to inject detailed information of MS images into the LrHS image, thereby generating the HrHS product. The latter one is to pansharpen the HS image by linearly combining MS bands to synthesize a HrHS band using regression techniques.

One representative technique that fuses the HS-MS images is the subspace-based approach, which are usually designed to enhance the spatial resolution of HS images by means of matrix factorization [14], [25]–[28] or Bayesian estimation [29], [30]. Kawakami et al. [14] first decomposed the LrHS image into a spectral dictionary and corresponding sparse representations via matrix factorization, and the HrHS image can be then formed by using shareable sparse coefficients estimated from the RGB observation with a sampled spectral dictionary. Akhtar et al. [26] jointly considered different physical properties of materials in the scene, e.g., sparsity, non-negativity, and spatial structure, to improve the quality of the HrHS image. Dong et al. [28] structurized the sparse coding model with the application to HS-SR. Yokoya’s algorithm [25] is designed to perform the HS-MS fusion via an effective couple matrix factorization approach, which can be well explained from the spectral unmixing perspective. Similarly, Lanaras et al. [27] further extended the [25] by jointly unmixing the input HS and MS data and fusing them in the latent subspace. In [29], the spectral dictionary is learned by using a parameter-free Bayesian model. Using the learned dictionary, the HrHS image can be reconstructed with potential sparse assumptions. Further, the same investigators used the Gaussian process as the priors on the basis of the Bayesian model to super-resolve the HS image [30]. Beyond matrix factorization, authors of [31] and [5] have attempted to model the HS-SR issue in tensor form to achieve better structural representations.

### B. Deep Learning based Models

These traditional methods have shown the competitive performance in the HS-MS fusion task, yet they, to a great extent, rely on cross-sensor calibration and strong hand-crafted priors. Inspired by the recent success of deep learning (DL) techniques, CNNs-based approaches have been garnering growing attention in the HS-SR task [16]–[20], [32]. Dian et al. [16] proposed a two-stage HS image pansharening framework by using CNNs-based prior training to refine the initialized fusion results obtained by traditional optimization methods. Qu et al. [17] trained an end-to-end unsupervised Dirichlet network to restore the HrHS image. However, the network needs to be alternatively optimized with given SRF
<figure><img src="image_4.png" /><figcaption>Fig. 2. An illustrative overview of the proposed DC-Net. It consists of two subnetworks: D-Net and C-Net. With the adversarial loss $L_{adv}^{XY}$ between the common content ($\bar{X}$ and $\bar{Y}$) of X and Y, the reconstruction loss $L_{rec}$, and the self-supervised module (i.e., S-Net), we can learn an end-to-end HS-SR network more effectively without any prior training, and better enhance detailed appearances.</figcaption></figure>

and PSF of hybrid cameras. To maximize the HrHS image reconstruction accuracy, Fu et al. [33] designed a CNNs-based HS-SR method by retrieving the optimal SRF from an external dataset. They further extended their work [34] to select the best SRF by the joint use of external and internal learning for HS image reconstruction at a more accurate level. Xie et al. [18] sought to open the “black-box” and presented an interpretable deep supervised model for the MS/HS fusion task. Yao et al. [19] attempted to explain an unsupervised deep super-resolution network from the spectral unmixing perspective by introducing cycle consistency and cross-attention mechanism. Zhu et al. [32] proposed a supervised deep residual network in a progressive way for the HS image super-resolution.

### III. DECOUPLED-AND-COUPLED NETWORKS

#### A. Overview

A large distribution gap between HS-MS images possibly caused by unknown sensor-specific information (e.g., HS-MS) and highly mixed spectral pixels has been shown to be a challenging and potential problem in the HS-SR task [15], [25], [27]. Focusing on it, we provide a point-to-point solution by designing a novel network architecture, i.e., DC-Net, which consists of two subnetworks. D-Net aims to decompose the original HS-MS data into the common content (or structure) and sensor-specific features, and implement crossover recombination on the sensor-specific information to improve the compatibility between HS-MS images for better subsequent fusion. With the input of recombined HS-MS images, C-Net performs the subpixel-level fusion by decomposing the mixed pixels into fractional coefficients (termed as abundances) and pure spectral signatures (termed as endmembers), yielding an interpretable CSU-guided HS-SR network. It should be noted that DC-Net (i.e., D-Net and C-Net) can be regarded as a novel progressive fusion framework from pixel- to subpixel-level, from the image- to feature-level. Further, a simple but effective self-supervised subnetwork (S-Net) in C-Net is devised to improve the quality of the HrHS product by correcting

the material correspondence between HS-MS images. Fig. 2 illustrates the architecture overview of the proposed DC-Net.

#### B. Problem Formulation

Let $\mathcal{X} \in \mathbb{R}^{h \times w \times L}$, $\mathcal{Y} \in \mathbb{R}^{H \times W \times l}$, and $\mathcal{Z} \in \mathbb{R}^{H \times W \times L}$ be LrHS with $hw$ pixels by $L$ bands, HrMS with $HW$ pixels by $l$ bands, and HrHS images with $HW$ pixels by $L$ bands, respectively, ($h < H$, $w < W$, $l < L$), then $\mathbf{X} \in \mathbb{R}^{L \times hw}$, $\mathbf{Y} \in \mathbb{R}^{l \times HW}$, and $\mathbf{Z} \in \mathbb{R}^{L \times HW}$ are defined as the unfolded matrices. The observation models for LrHS and HrMS images from the HrHS image can be written as

$$ 
\begin{aligned}
\mathbf{X} &= \mathbf{Z}\mathbf{R} + \mathbf{N}_x, \\
\mathbf{Y} &= \mathbf{H}\mathbf{Z} + \mathbf{N}_y,
\end{aligned} 
\qquad (1) $$

where $\mathbf{H} \in \mathbb{R}^{l \times L}$ and $\mathbf{R} \in \mathbb{R}^{HW \times hw}$ denote the SRF and PSF to degrade the spectral and spatial resolutions of $\mathbf{Z}$, respectively, and $\mathbf{N}_x$ and $\mathbf{N}_y$ are the observed noises.

Ideally, the observations in Eq. (1) can be further unfolded by using spectral mixing model as

$$ 
\begin{aligned}
\mathbf{X} &= \mathbf{S}\mathbf{A}\mathbf{R} + \mathbf{N}_x = \mathbf{S}\tilde{\mathbf{A}} + \mathbf{N}_x, \\
\mathbf{Y} &= \mathbf{H}\mathbf{S}\mathbf{A} + \mathbf{N}_y = \tilde{\mathbf{S}}\mathbf{A} + \mathbf{N}_y,
\end{aligned} 
\qquad (2) $$

where $\mathbf{S} \ge 0$ and $\mathbf{A} \ge 0$ denote a collection of pure spectral signatures (i.e., endmembers) and corresponding fractional coefficients (i.e., abundances) that meet the sum-to-one constraint (i.e., $1^\top \mathbf{A} = 1^\top$), respectively, while $\tilde{\mathbf{S}}$ and $\tilde{\mathbf{A}}$ can be explained as the degraded $\mathbf{S}$ and $\mathbf{A}$ in spectral and spatial domains with the use of SRF and PSF. Thus, estimating $\mathbf{Z}$ is equivalent to finding $\mathbf{S}$ and $\mathbf{A}$, i.e., $\mathbf{Z} = \mathbf{S}\mathbf{A}$, by solving the problem (2).

The model (2) has been proven to be effective for the problem of mixed pixels in the HS-MS fusion [15], [17], [19], [25], [27], [31]. However, it seldom considers the high coupling between pixels in the same data source. For this reason, we learn the mappings ($f_x$ and $f_y$) for transforming $\mathbf{X}$ and $\mathbf{Y}$ to a latent image space (e.g., $\bar{\mathbf{X}}$, $\bar{\mathbf{Y}}$), where the pixel information can be easily separated between the same
<figure><img src="image_5.png" /><figcaption>Fig. 3. An example for SRF (i.e., H) in RGB bands.</figcaption></figure>

data source and better fused across different data sources. Therefore, the resulting model is

$$ 
\begin{aligned}
\bar{X} &= f_x(\mathbf{X}) = f_x(\mathbf{S}\tilde{\mathbf{A}} + \mathbf{N}_x), \\
\bar{Y} &= f_y(\mathbf{Y}) = f_y(\tilde{\mathbf{S}}\mathbf{A} + \mathbf{N}_y).
\end{aligned}
\quad (3) $$

In the following, we will build the DC-Net architecture step-by-step to perform the HS-MS fusion in the model (3).

### C. Decoupled Network (D-Net)

Owing to the powerful image-to-image transformation ability of GANs [35]–[37], we develop the D-Net to decouple the input images and recombine them (see Fig. 2), making it possible for the recombined images to be fused more sufficiently. D-Net embeds the HS-MS images onto a common content space $\mathcal{C}$ and sensor-specific spaces, e.g., $\mathcal{S}_\mathcal{X}$ and $\mathcal{S}_\mathcal{Y}$. This process can be performed by cross-sensor (or domain-common) encoders $\{E_\mathcal{X}^C, E_\mathcal{Y}^C\}$, sensor-specific encoders $\{E_\mathcal{X}^S, E_\mathcal{Y}^S\}$, generators $\{G_\mathcal{X}, G_\mathcal{Y}\}$, and a domain-common discriminator $D_{\mathcal{adv}}^C$. Given the input data ($\mathbf{X}$ and $\mathbf{Y}$), we then have the following transformations:

$$ 
\begin{aligned}
\{\mathbf{X}^C, \mathbf{X}^S\} &= \{E_\mathcal{X}^C(\mathbf{X}), E_\mathcal{X}^S(\mathbf{X}_\uparrow)\}, \\
\{\mathbf{Y}^C, \mathbf{Y}^S\} &= \{E_\mathcal{Y}^C(\mathbf{Y}), E_\mathcal{Y}^S(\mathbf{Y}_\downarrow)\},
\end{aligned}
\quad (4) $$

where $\mathbf{X}^C \in \mathcal{C}$ and $\mathbf{Y}^C \in \mathcal{C}$ are the common information between $\mathbf{X}$ and $\mathbf{Y}$, and $\mathbf{X}^S \in \mathcal{S}_\mathcal{X}$ and $\mathbf{Y}^S \in \mathcal{S}_\mathcal{Y}$ are the domain-specific information.

By interactively recombining the common and sensor-specific information, we can further obtain transformed images, i.e., $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{Y}}$, via generators ($\{G_\mathcal{X}, G_\mathcal{Y}\}$). This process can be formulated by $\tilde{\mathbf{X}} = G_\mathcal{X}(\mathbf{X}^C, \mathbf{Y}^S)$ and $\tilde{\mathbf{Y}} = G_\mathcal{Y}(\mathbf{Y}^C, \mathbf{X}^S)$. Such recombined images provide higher affinity between different sources (e.g., HS-MS), thereby yielding a greater potential for follow-up fusion.

### D. Coupled Network (C-Net)

Linking to the D-Net, C-Net unmixes the recombined HS and MS images into endmembers ($\mathbf{S}$, $\tilde{\mathbf{S}}$) and abundances ($\tilde{\mathbf{A}}$, $\mathbf{A}$). As shown in Fig. 2, C-Net is a coupled convolutional autoencoder, which consists of two encoders ($f_u$ and $g_u$) and two decoders ($f_v$ and $g_v$) for $\tilde{\mathbf{X}}$ and $\tilde{\mathbf{Y}}$, respectively. This can be formulated accordingly as

$$ 
\begin{aligned}
\hat{\mathbf{X}} &= \mathbf{S}\tilde{\mathbf{A}} = f(\tilde{\mathbf{X}}) = f_v(f_u(\tilde{\mathbf{X}}; \mathbf{W}_{f,u}); \mathbf{W}_{f,v}), \\
\hat{\mathbf{Y}} &= \tilde{\mathbf{S}}\mathbf{A} = g(\tilde{\mathbf{Y}}) = g_v(g_u(\tilde{\mathbf{Y}}; \mathbf{W}_{g,u}); \mathbf{W}_{g,v}).
\end{aligned}
\quad (5) $$

$\mathbf{W}$ is the weights of C-Net, where $\mathbf{W}_{f,v}$ and $\mathbf{W}_{g,v}$ are specified to be a 1×1 linear convolutional kernel, which can be well

<figure><img src="image_6.png" /><figcaption>Fig. 4. An illustration of S-Net module.</figcaption></figure>

explained as endmembers, i.e., $\mathbf{S}$ and $\tilde{\mathbf{S}}$. Note that we adopt the Clamp function [38] to meet the abundance non-negative constraint (ANC), while the abundance sum-to-one constraint (ASC), i.e., $\mathbf{1}^\top\mathbf{A} = \mathbf{1}^\top$, can be guaranteed in the form of regularization. Using Eq. (5), the to-be-estimated HrHS image $\hat{\mathbf{Z}}$ can be obtained by $\hat{\mathbf{Z}} = \mathbf{S}\mathbf{A} = f_v(g_u(\tilde{\mathbf{Y}}; \mathbf{W}_{g,u}); \mathbf{W}_{f,v})$.

**Cycle consistency.** Coupled factors, i.e., SRF ($\mathbf{H}$) and PSF ($\mathbf{R}$), play a dominant role in the performance improvement of C-Net. Fig. 3 shows a SRF example w.r.t RGB bands. Unlike previous methods that assume the two functions to be known, we attempt to automatically learn them by designing a cycle consistency mechanism in networks, enabling it to be more applicable in reality. For simplicity, the cycle can be written as $(\mathbf{X}, \mathbf{Y}) \xrightarrow{\text{D-Net}} (\tilde{\mathbf{X}}, \tilde{\mathbf{Y}}) \xrightarrow{\text{C-Net}} \hat{\mathbf{Z}} \xrightarrow{(\mathbf{H}, \mathbf{R})} (\mathbf{X}, \mathbf{Y})$. More specifically, given the spectrum of the i-th pixel in $\mathbf{Z}$ (denoted as $z_i$) and the corresponding spectrum of the i-th pixel and the j-th channel in $\mathbf{Y}$ (denoted as $y_{i,j}$), the degradation process can be then expressed as

$$ y_{i,j} = \int_{\phi} \frac{h_j(\eta)}{N_h} z_i(\eta) d\eta, \quad (6) $$

where $\phi$ denotes the support set w.r.t. the wavelength $\eta$, $\int h_j(\eta)d\eta$ is the continuous representation of $\mathbf{H}$, and $N_h$ is a constant for scaled normalization. In our case, $\mathbf{H}$ can be approximated using a set of 1 × 1 × L convolutional kernels, e.g., $\mathbf{w}_j$, in a discretized way, i.e.,

$$ y_{i,j} = \mathbf{z}_j \otimes \mathbf{w}_j = \sum_{\phi} \frac{\mathbf{w}_j(\eta)}{N_w} z_i(\eta). \quad (7) $$

Similarly, spatial downsampling process performed by the PSF ($\mathbf{R}$) can be also simulated by means of convolutional operators with same scaling kernel and stride sizes.

As a result, the cycle consistency for both spatial and spectral degradation can be well organized in the C-Net as

$$ \mathbf{X} = \hat{\mathbf{Z}}\mathbf{R}, \quad \mathbf{Y} = \mathbf{H}\hat{\mathbf{Z}}, \quad (8) $$

which is a good fit for the observation models in Eq. (1) and helps us form a closed cycle chain.
### E. Self-Supervised Subnetwork (S-Net)

In C-Net, the spectral unmixing process for HS and MS images is relatively independent. This might lead to chaotic semantic correspondences on abundance maps (or endmembers) of HS-MS images, e.g., between A and $\tilde{A}$ or $\tilde{S}$ and S. To this end, we design an effective self-supervised learning module (S-Net) in C-Net to obtain more plausible abundance maps and endmembers, further yielding a more accurate HS reconstruction result. Fig. 4 illustrates the S-Net.

S-Net aims at correcting the correspondence of abundance maps of HS-MS images by learning high-level semantic alignment or consistency (in both order and category of materials), enabling a one-to-one match between A and $\tilde{A}$. We call it as material or semantic consistency. The abundance maps of the same materials should be attracted to each other (as positive samples); otherwise, they are repelled (as negative samples). Considering the computational cost (since hundreds of abundance maps and spectral bundles usually need to be considered), we employ the same grouping strategy on A and $\tilde{A}$ to make a group-to-group match. A and $\tilde{A}$ can be grouped as

$$ 
\begin{aligned}
\mathbf{A} &= \left[ \underbrace{\mathbf{A}_1, \dots, \mathbf{A}_p}_{\Phi_{A,1}}, \dots, \underbrace{\mathbf{A}_k, \dots, \mathbf{A}_p}_{\Phi_{A,p}}, \dots, \underbrace{\dots, \mathbf{A}_n}_{\Phi_{A,m}} \right], \\
\tilde{\mathbf{A}} &= \left[ \underbrace{\tilde{\mathbf{A}}_1, \dots, \tilde{\mathbf{A}}_p}_{\Phi_{\tilde{A},1}}, \dots, \underbrace{\tilde{\mathbf{A}}_k, \dots, \tilde{\mathbf{A}}_p}_{\Phi_{\tilde{A},p}}, \dots, \underbrace{\dots, \tilde{\mathbf{A}}_n}_{\Phi_{\tilde{A},m}} \right],
\end{aligned} 
\quad (9) $$

where $\Phi_{A,p}$ and $\Phi_{\tilde{A},q}$ denote the grouping sets w.r.t. A and $\tilde{A}$ for $\forall p, q \in \{1, \dots, m\}$, respectively. We experimentally set the grouping parameter $m$ to be 8. Furthermore, the representation vectors of $\Phi_{A,p}$ and $\Phi_{\tilde{A},q}$, denoted as $z$, can be obtained via a two-stream encoder or CNN, e.g., $f_{\Phi_A}$ and $f_{\Phi_{\tilde{A}}}$, which both consists of two blocks, i.e., [conv-relu-pool-fc]. The main difference lies in the pooling size and stride for $f_{\Phi_A}$ larger than those for $f_{\Phi_{\tilde{A}}}$ to guarantee the same size for final output features. We then have $z_{A,p} = f_{\Phi_A}(\Phi_{A,p})$ and $z_{\tilde{A},q} = f_{\Phi_{\tilde{A}}}(\Phi_{\tilde{A},q})$. In our S-Net, only when $p = q$, $z_{\tilde{A},q}$ is the positive sample of $z_{A,p}$; otherwise ($p \neq q$), they are the negative sample pair.

### F. Network Training

**Loss functions.** As shown in Fig. 2, there are several important loss functions that need to be carefully considered.

In D-Net, the adversarial loss between the common content of X and Y can be expressed as

$$ 
\begin{aligned}
\mathcal{L}_{adv} ={}& \mathbb{E}_x \left[ \frac{1}{2} \log D^C(\mathbf{X}^C) + \frac{1}{2} \log(1 - D^C(\mathbf{X}^C)) \right] \\
& + \mathbb{E}_y \left[ \frac{1}{2} \log D^C(\mathbf{Y}^C_{\downarrow}) + \frac{1}{2} \log(1 - D^C(\mathbf{Y}^C_{\downarrow})) \right],
\end{aligned} 
\quad (10) $$

where $\mathbf{X}^C = E_X^C(\mathbf{X})$ and $\mathbf{Y}^C = E_Y^C(\mathbf{Y})$.

The reconstruction loss stemmed from coupled convolutional autoencoders and the cycle consistency can be computed by means of Eqs. (5) and (8):

$$ 
\begin{aligned}
\mathcal{L}_{rec} ={}& \|\hat{\mathbf{X}} - \mathbf{X}\|_1 + \|\hat{\mathbf{Y}} - \mathbf{Y}\|_1 + \|\hat{\mathbf{X}} - \mathbf{Z}_{\downarrow,R}\|_1 \\
& + \|\hat{\mathbf{Y}} - \mathbf{Z}_{\downarrow,H}\|_1 + \|\bar{\mathbf{X}} - \mathbf{X}\|_1 + \|\bar{\mathbf{Y}} - \mathbf{Y}\|_1,
\end{aligned} 
\quad (11) $$

where the $l_1$-norm measure is applied to enhance the detailed perception in image reconstruction [39], and $\hat{\mathbf{Z}}_{\downarrow,R}$ and $\hat{\mathbf{Z}}_{\downarrow,H}$ denote $\hat{\mathbf{Z}}R$ and $\hat{\mathbf{H}}\hat{\mathbf{Z}}$, respectively.

Moreover, the ASC is satisfied using the following loss:

$$ \mathcal{L}_{ASC} = \|\mathbf{1}^{\top} - \mathbf{1}^{\top} \mathbf{A}\|_1 + \|\mathbf{1} - \mathbf{1}^{\top} \tilde{\mathbf{A}}\|_1. \quad (12) $$

In S-Net, the InfoNCE loss [40] is optimized to measure the distances between positive and negative samples:

$$ \mathcal{L}_{self} = -\mathbb{E}_{\Phi} \left[ \log \frac{\exp(f_{\Phi}(\Phi_{A,p})^{\top} f_{\Phi}(\Phi_{\tilde{A},q}^{+}))}{\sum_{q=1}^{m} \exp(f_{\Phi}(\Phi_{A,p})^{\top} f_{\Phi}(\Phi_{\tilde{A},q}))} \right], \quad (13) $$

where $\Phi_{A,p}$ and $\Phi_{\tilde{A},q}^{+}$ denote the anchor sample and the positive sample, respectively.

To sum up, our DC-Net is trained by minimizing the following overall loss parameterized by the set $\{\alpha, \beta, \gamma\}$:

$$ \mathcal{L} = \mathcal{L}_{rec} + \alpha \mathcal{L}_{adv} + \beta \mathcal{L}_{ASC} + \gamma \mathcal{L}_{self}. \quad (14) $$

**Implementation details.** Our networks are implemented on the PyTorch platform, and the Adam [41] optimizer is used to train the networks for 10000 epochs with a batch size of 1. The base learning rate starts with 0.005, which can be gradually updated by a linear decay schedule [42]. We initialize all convolutional layers in our networks with Kaiming initialization [43]. Furthermore, the hyper-parameters are experimentally determined using a grid search, and early stopping strategy is considered in the network training when the validation loss fails to decrease.

## IV. EXPERIMENTAL RESULTS

In this section, we first review the datasets and experimental setting. Next, we verify our networks with extensive ablation studies. Last, the performance evaluation on simulated and real datasets is conducted to show the superiority of DC-Net in comparison with state-of-the-art methods.

### A. Dataset and Setup

Two public HS-MS datasets are used to evaluate the performance of DC-Net, i.e., CAVE dataset (indoor) [44] and Chikusei dataset (remote sensing) [15].

The CAVE dataset consists of 32 indoor HrHS images with the size of 512 × 512 × 31, covering the spectral wavelength from 400nm to 700nm. The Chikusei dataset acquired by VNIR-C sensor comprises of 2517 × 2335 pixels and 128 spectral bands. We randomly crop the scene into 12 non-overlapped subimages with spatial size of 576×448. Moreover, we divide the two datasets into training, validation, and testing sets in the proportions of 8:8:16 and 4:4:4, respectively, where training and validation sets are used to determine optimal hyperparameters.

For the two datasets, the HrMS images are generated by degrading the HrHS images in spectral domain with SRFs, while the LrHS images can be also simulated by spatially downsampling the HrHS images with the Wald's protocol [45] of a 32 sampling ratio. In addition, the Nikon D700 sensor and Landsat-8 MS sensor [46] are selected as the SRFs for the CAVE and Chikusei datasets, respectively.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Visualization of super-resolved HS images obtained by all compared methods (CAVE: *chart and stuffed toy*), where a ROI zoomed in 4 times (bottom-right) and the difference image with GT (bottom-left) are highlighted.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6. Visualization of super-resolved HS images obtained by all compared methods (CAVE: *stuffed toys*), where a ROI zoomed in 4 times (bottom-right) and the difference image with GT (bottom-left) are highlighted.</figcaption></figure>

**Evaluation Metrics.** We evaluate the fusion performance quantitatively in terms of five widely-used picture quality indices (PQIs): peak signal-to-noise ratio (PSNR), spectral angle mapper (SAM) [47], *erreur relatif globale adimensionnelle de synthèse* (ERGAS) [48], structure similarity (SSIM) [49], and universal image quality index (UIQI) [50].

## B. Comparison with State-of-the-arts

We evaluate the performance of DC-Net both quantitatively and qualitatively in comparison with state-of-the-art (SOTA) models in the HS-SR task, including traditional SOTA methods: BSR [29]<fnref n="1" />, CSU [27]², NSSR [28]³, NLSTF [31]⁴, and DL-based SOTA methods: uSDN [17]⁵, MHF-Net [18]⁶, GDD

[20]<fnref n="7" />, CUCaNet [19]⁸, as well as the DC-Net without S-Net module. We maintain the same experimental configurations for all compared methods as in their original literature as much as possible.

1) *Experiments on CAVE Data:* Table I lists the average results over 16 testing images on CAVE dataset to make quantitative comparison between all compared SOTA methods and the DC-Net and DC-Net-S in terms of five PQIs.

Overall, BSR, as a pioneer work in HS-SR, reconstructs the high-quality HS image from the Bayesian statistic point of view and achieves competitive performance. Despite relatively lower PSNR and SSIM (cf. BSR), CSU can obtain better POIs in SAM, ERGAS, and UQI, owing in large part to the use of coupled spectral unmixing mechanism. Beyond the CSU, NLSTF fuses HS and MS images in a coupled tensor form with non-local spatial information preservation, yielding better performance in all indices except the UQI. By virtue of non-negativity and structured modeling, NSSR

<footnote n="1">¹https://github.com/junjun-jiang/Hyperspectral-Image-Super-Resolution-Benchmark
²https://github.com/lanaa/SupResPALM
³http://see.xidian.edu.cn/faculty/wsdong
⁴https://github.com/renweidian/NLSTF
⁵https://github.com/aicip/uSDN
⁶https://github.com/XieQi2015/MHF-net</footnote>

<footnote n="7">⁷https://github.com/tuezato/guided-deep-decoder
⁸https://github.com/danfenghong/ECCV2020_CUCaNet</footnote>
<table><caption>TABLE I
QUANTITATIVE AVERAGE PERFORMANCE OF SOTA SUPER-RESOLUTION METHODS OVER 16 TESTING IMAGES IN TERMS OF FIVE DIFFERENT PQIS ON CAVE DATASET.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Ref.</th><th colspan="5">Metric</th></tr><tr><th>PSNR</th><th>SAM</th><th>ERGAS</th><th>SSIM</th><th>UQI</th></tr></thead><tbody><tr><td>BSR</td><td>CVPR'15 [29]</td><td>31.08</td><td>13.52</td><td>0.72</td><td>0.901</td><td>0.894</td></tr><tr><td>CSU</td><td>ICCV'15 [27]</td><td>29.69</td><td>13.13</td><td>0.70</td><td>0.891</td><td>0.900</td></tr><tr><td>NSSR</td><td>TIP'16 [28]</td><td>33.93</td><td>10.87</td><td>0.61</td><td>0.925</td><td>0.914</td></tr><tr><td>NLSTF</td><td>CVPR'17 [31]</td><td>30.31</td><td>12.29</td><td>0.69</td><td>0.914</td><td>0.896</td></tr><tr><td>uSDN</td><td>CVPR'18 [17]</td><td>35.60</td><td>9.97</td><td>0.53</td><td>0.934</td><td>0.921</td></tr><tr><td>MHF-Net</td><td>CVPR'19 [18]</td><td>37.41</td><td>7.33</td><td>0.45</td><td>0.965</td><td>0.958</td></tr><tr><td>GDD</td><td>ECCV'20 [20]</td><td>37.06</td><td>6.71</td><td>0.40</td><td>0.971</td><td>0.960</td></tr><tr><td>CUCaNet</td><td>ECCV'20 [19]</td><td>37.29</td><td>7.03</td><td>0.46</td><td>0.960</td><td>0.951</td></tr><tr><td>DC-Net</td><td>–</td><td>38.12</td><td>6.48</td><td>0.39</td><td>0.973</td><td>0.977</td></tr><tr><td>DC-Net-S</td><td>–</td><td><b>38.68</b></td><td><b>6.15</b></td><td><b>0.34</b></td><td><b>0.978</b></td><td><b>0.980</b></td></tr><tr><td>Ideal Value</td><td>–</td><td>↑</td><td>↓ 0</td><td>↓ 0</td><td>↑ 1</td><td>↑ 1</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Visualization (R: 63, G: 35, B: 11) of super-resolved HS images obtained by all compared methods on Chikusei data, where a ROI zoomed in 4 times and the difference image with GT are highlighted for more detailed observation.</figcaption></figure>

outperforms other traditional SOTA methods in HrHS image reconstruction. As an early-stage DL-based method, the fusion performance of uSDN is moderately inferior to that of other DL-based competitors, but is clearly higher than that of those non-DL algorithms. It is more noteworthy that the results obtained by unsupervised GDD and CUCaNet are basically comparable to that of the supervised MHF-Net (even better in certain indices). DC-Net behaves superiorly compared to existing SOTA DL approaches, implying that the decoupled-and-coupled strategy creates the “one plus one greater than two” effect. Further, S-Net module can learn high-level semantic consistency from abundance maps and embed interpretable and physically meaningful information into the DC-Net, enabling the best performance (i.e., DC-Net-S).

Figs. 5 and 6 visualize the super-resolved HS image in the

11-th band (ca. 500nm) for *chart and stuffed toy*. Further, two regions of interest (ROIs) are selected to highlight the visually detailed differences of all compared methods. By and large, DL-based models outperform evidently compared to traditional methods, particularly in coarser-grained structure of objects. The results of the proposed methods are superior to other unsupervised ones (e.g., uSDN, GDD, CUCaNet), while DC-Net-S performs better than DC-Net in terms of detailed appearances. Despite similar visual results (cf. MHF-Net), our networks, either DC-Net or DC-Net-S, are capable of better recovering textural details and color cues that approximate to ground truth (GT).

2) *Experiments on Chikusei Data:* Quantitative performance comparison on Chikusei dataset is given in Table II, where there is a basically identical trend with that on CAVE
<figure><img src="image_10.png" /><figcaption>Fig. 8. Visual evaluation on CAVE and Chikusei datasets, respectively. The GT, the residual maps between GT maps and super-solved HS results obtained by compared methods (e.g., NSSR, uSDN, MHF-Net, CUCaNet) and our proposed DC-Net and DC-Net-S. The RMSE values of HS images for these methods along the spectra (covering the wavelength) are also given corresponding to the six studied scenes.</figcaption></figure>

dataset (see Table 1). Remote sensing images (due to the lower spatial resolution) are not as complicated as indoor images, all these methods, therefore, tend to super-resolve HS images with higher quality on Chikusei dataset (cf. CAVE). Significantly, the performance of our DC-Net-S is still superior to other competitors w.r.t. all PQIs, demonstrating its superiority in the HS-SR task. However, the most noteworthy point lies in that MHF-Net obtains competitive results (fully comparable to DC-Net without S-Net module), since its accuracy relies greatly on supervised training on plenty of sample pairs. Fig. 7 gives a scene example and shows the composite HrHS images obtained by all competing methods with a ROI zoomed in 4 times for better observing visual differences.

3) Visual Quality Evaluation: To evaluate the visual quality of HS super-solved results more finely, we select several representative images from CAVE (four scenes) and Chikusei (two scenes) datasets, respectively, and show the corresponding

residual maps between GT maps and super-solved HS results obtained by different compared methods, e.g., NSSR, uSDN, MHF-Net, CUCaNet, and our proposed DC-Net and DC-Net-S, in Fig. 8. Furthermore, the RMSE values of HS images in each band are shown along spectra corresponding to the aforementioned six scenes. Intuitively, our methods perform better than other competitors from the perspective of residual maps, which demonstrates that our methods can provide finer spatial details. More importantly, the stability and generalization ability of our DC-Nets are higher than compared networks in different scenes, showing the effectiveness to a great extent. The band-wise RMES results obtained by our models are obviously similar and closer to the GT in the six investigated scenes. This, to some extent, indicates that the sequentially spectral information can be well recovered and retained by our approaches.
<table><caption>TABLE II
QUANTITATIVE PERFORMANCE COMPARISON BETWEEN THE SAME SOTA SUPER-RESOLUTION METHODS IN TERMS OF FIVE DIFFERENT PQIS ON CHIKUSEI DATASET.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Ref.</th><th colspan="5">Metric</th></tr><tr><th>PSNR</th><th>SAM</th><th>ERGAS</th><th>SSIM</th><th>UQI</th></tr></thead><tbody><tr><td>BSR</td><td>CVPR'15 [29]</td><td>38.13</td><td>4.54</td><td>0.60</td><td>0.948</td><td>0.963</td></tr><tr><td>CSU</td><td>ICCV'15 [27]</td><td>36.96</td><td>5.20</td><td>0.73</td><td>0.898</td><td>0.937</td></tr><tr><td>NSSR</td><td>TIP'16 [28]</td><td>42.30</td><td>3.12</td><td>0.48</td><td>0.970</td><td>0.984</td></tr><tr><td>NLSTF</td><td>CVPR'17 [31]</td><td>37.04</td><td>6.11</td><td>0.70</td><td>0.901</td><td>0.925</td></tr><tr><td>uSDN</td><td>CVPR'18 [17]</td><td>40.02</td><td>2.96</td><td>0.41</td><td>0.964</td><td>0.981</td></tr><tr><td>MHF-Net</td><td>CVPR'19 [18]</td><td>42.43</td><td>3.01</td><td>0.44</td><td>0.977</td><td>0.982</td></tr><tr><td>GDD</td><td>ECCV'20 [20]</td><td>41.58</td><td>3.04</td><td>0.43</td><td>0.974</td><td>0.984</td></tr><tr><td>CUCaNet</td><td>ECCV'20 [19]</td><td>41.79</td><td>2.84</td><td>0.40</td><td>0.975</td><td>0.988</td></tr><tr><td>DC-Net</td><td>–</td><td>42.08</td><td>2.78</td><td>0.38</td><td>0.977</td><td>0.989</td></tr><tr><td>DC-Net-S</td><td>–</td><td><b>42.56</b></td><td><b>2.55</b></td><td><b>0.34</b></td><td><b>0.979</b></td><td><b>0.992</b></td></tr><tr><td>Ideal Value</td><td>–</td><td>↑</td><td>↓ 0</td><td>↓ 0</td><td>↑ 1</td><td>↑ 1</td></tr></tbody></table>

<table><caption>TABLE III
ABLATION ANALYSIS OF THE PROPOSED DC-NET WITH A COMBINATION OF DIFFERENT MODULES ON THE CAVE DATASET.</caption><thead><tr><th rowspan="2">Method</th><th colspan="5">Module</th><th colspan="2">Metric</th></tr><tr><th>ANC</th><th>ASC</th><th>D-Net</th><th>C-Net</th><th>S-Net</th><th>PSNR</th><th>SAM</th></tr></thead><tbody><tr><td>C-Net</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>32.56</td><td>14.80</td></tr><tr><td>C-Net</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>36.69</td><td>7.43</td></tr><tr><td>C-Net</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>36.97</td><td>6.84</td></tr><tr><td>DC-Net</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✗</td><td>38.36</td><td>4.77</td></tr><tr><td>DC-Net-S</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td><b>39.15</b></td><td><b>4.51</b></td></tr><tr><td>Ideal Value</td><td>–</td><td>–</td><td>–</td><td>–</td><td>–</td><td>↑</td><td>↓ 0</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 9. Visualization of super-resolved HrHS fused by WV-3 HrMS (image courtesy Maxar) and Hyperion LrHS using our method (real data of Cuprite: three sub-regions).</figcaption></figure>

### C. Evaluation on Real Data

To evaluate the performance of our DC-Net effectively, we collect a set of real and registered MS and HS scenes captured by WorldView-3 and Hyperion satellite missions, respectively. Three sub-regions are selected to visualize the quality comparison of composite HrHS products, as shown in

Fig. 9. Each sub-region consists of HrMS and LrHS images with the size of 640 × 640 pixels with 8 bands at a 7.5m ground sampling distance (GSD) and 160 × 160 pixels with 167 spectral channels at a GSD of 30m, respectively, while the real HrHS image is not available. In Fig. 9, both DC-Net and DC-Net-S can restore high resolution HrHS results. Despite clear results obtained by the two methods, the latter gives a higher degree of visual verisimilitude. Compared to the DC-Net, the structural and textural appearances of the DC-Net-S sit at the same level as the HrMS image and even beyond, while its color and brightness are closer to that of the LrHS image, showing the superiority of the proposed methods in the HS-SR task.

### D. Model Analysis

1) **Ablation Studies:** We investigate the performance gain by stepwise adding different modules (or subnetworks, i.e., D-Net and S-Net) in networks. We also study the importance of two physical constraints (ANC and ASC) related to spectral unmixing model and their effects on the quality of HS-SR. To verify the effectiveness of the ablation analysis, we report the average results in terms of PSNR and SAM indices over 8 images on the validation set of the CAVE dataset by maximizing the stepwise performance using optimal parameters, as listed in Table III.

In detail, C-Net without ANC and ASC yields relatively poor performance, which to some extent indicates the impor-
<table><caption>TABLE IV
PARAMETER SIZE AND RUNNING TIME OF OUR DC-NET-S.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Size (MB)</th><th colspan="3">Average Run Time (min.)</th></tr><tr><th>CAVE</th><th>Chikusei</th><th>Cuprite</th></tr></thead><tbody><tr><td>DC-Net-S</td><td>3.43</td><td>101</td><td>312</td><td>156</td></tr></tbody></table>

tance of the two important physical constraints in the spectral unmixing-inspired HS-SR network. By turning on the ANC, the fusion results of C-Net increase dramatically (beyond 4 PNSR value and around halved SAM value), What is better still, the joint use of ANC and ASC can further bring the performance improvement in both PSNR and SAM compared to ANC used alone in C-Net. As expected, DC-Net after plugging the D-Net performs observably better than C-Net at an increase of about 1.5 PSNR value and a decrease of just over 2 SAM value. More remarkably, the self-supervised module (S-Net) is capable of breaking through the existing “bottleneck” in DC-Net by enforcing the semantic consistency on abundance maps (A and Ā).

2) Computational Analysis: We list the size of network parameters (i.e., DC-Net-S) and the inference time per image on a PC with one NVIDIA GeForce GTX 1080Ti GPU in Table. IV. Our network parameters only have 3.43MB, and since our method is unsupervised, the inference time is also desirable and acceptable in such a PC setting.

## V. CONCLUSION

In this paper, we propose a novel subpixel-level HS-SR framework, i.e., DC-Net, by fully considering the effects of affinity of the same data sources and exclusivity across HS-MS data and mixed pixels within one pixel. Inspired by the spectral unmixing, DC-Net is capable of utilizing the intrinsic properties of HS-MS images effectively for the fusion task, which is more applicable to various real cases. We further optimize the network performance via devising an effective self-supervised module. Extensive experiments conducted on simulated and real data show the superiority of DC-Net in HS-SR over current SOTA fusion methods. We found, however, that there exists spectral degradation in the HS super-resolution products of the real data. As a result, we will pay more attention to the development of the new module or the use of the new network architecture to preserve the spectral sequentiality as much as possible in the process of improving the resolution.

## REFERENCES

[1] B. Arad and O. Ben-Shahar, “Sparse recovery of hyperspectral signal from natural rgb images,” in *Proceedings of the European Conference on Computer Vision (ECCV)*, pp. 19–34, 2016.

[2] W. He, N. Yokoya, and X. Yuan, “Fast hyperspectral image recovery of dual-camera compressive hyperspectral imaging via non-iterative subspace-based fusion,” *IEEE Transactions on Image Processing*, vol. 30, pp. 7170–7183, 2021.

[3] S. Sarkar and R. R. Sahay, “A non-local superpatch-based algorithm exploiting low rank prior for restoration of hyperspectral images,” *IEEE Transactions on Image Processing*, vol. 30, pp. 6335–6348, 2021.

[4] J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding, and J. Paisley, “Pannet: A deep network architecture for pan-sharpening,” in *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, pp. 5449–5457, 2017.

[5] L. Zhang, W. Wei, C. Bai, Y. Gao, and Y. Zhang, “Exploiting clustering manifold structure for hyperspectral imagery super-resolution,” *IEEE Transactions on Image Processing*, vol. 27, no. 12, pp. 5969–5982, 2018.

[6] Y. Xu, Z. Wu, J. Chanussot, and Z. Wei, “Nonlocal patch tensor sparse representation for hyperspectral image super-resolution,” *IEEE Transactions on Image Processing*, vol. 28, no. 6, pp. 3034–3047, 2019.

[7] J. Xue, Y.-Q. Zhao, Y. Bu, W. Liao, J. C.-W. Chan, and W. Philips, “Spatial-spectral structured sparse low-rank representation for hyperspectral image super-resolution,” *IEEE Transactions on Image Processing*, vol. 30, pp. 3084–3097, 2021.

[8] W. Dong, C. Zhou, F. Wu, J. Wu, G. Shi, and X. Li, “Model-guided deep hyperspectral image super-resolution,” *IEEE Transactions on Image Processing*, vol. 30, pp. 5754–5768, 2021.

[9] A. Chakrabarti and T. Zickler, “Statistics of real-world hyperspectral images,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 193–200, 2011.

[10] N. M. Nasrabadi, “Hyperspectral target detection: An overview of current and future challenges,” *IEEE Signal Processing Magazine*, vol. 31, no. 1, pp. 34–44, 2013.

[11] G. Tochon, J. Chanussot, M. Dalla Mura, and A. L. Bertozzi, “Object tracking by hierarchical decomposition of hyperspectral video sequences: Application to chemical gas plume tracking,” *IEEE Transactions on Geoscience and Remote Sensing*, vol. 55, no. 8, pp. 4567–4585, 2017.

[12] X. Wu, D. Hong, J. Tian, J. Chanussot, W. Li, and R. Tao, “Orsim detector: A novel object detection framework in optical remote sensing imagery using spatial-frequency channel features,” *IEEE Transactions on Geoscience and Remote Sensing*, vol. 57, no. 7, pp. 5146–5158, 2019.

[13] F. Xiong, J. Zhou, and Y. Qian, “Material based object tracking in hyperspectral videos,” *IEEE Transactions on Image Processing*, vol. 29, pp. 3719–3733, 2020.

[14] R. Kawakami, Y. Matsushita, J. Wright, M. Ben-Ezra, Y.-W. Tai, and K. Ikeuchi, “High-resolution hyperspectral imaging via matrix factorization,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 2329–2336, 2011.

[15] N. Yokoya, C. Grohnfeldt, and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” *IEEE Geoscience and Remote Sensing Magazine*, vol. 5, no. 2, pp. 29–56, 2017.

[16] R. Dian, S. Li, A. Guo, and L. Fang, “Deep hyperspectral image sharpening,” *IEEE Transactions on Neural Networks and Learning Systems*, vol. 29, no. 11, pp. 5345–5355, 2018.

[17] Y. Qu, H. Qi, and C. Kwan, “Unsupervised sparse dirichlet-net for hyperspectral image super-resolution,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 2511–2520, 2018.

[18] Q. Xie, M. Zhou, Q. Zhao, D. Meng, W. Zuo, and Z. Xu, “Multispectral and hyperspectral image fusion by ms/hs fusion net,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 1585–1594, 2019.

[19] J. Yao, D. Hong, J. Chanussot, D. Meng, X. Zhu, and Z. Xu, “Cross-attention in coupled unmixing nets for unsupervised hyperspectral super-resolution,” in *Proceedings of the European Conference on Computer Vision (ECCV)*, pp. 208–224, 2020.

[20] T. Uezato, D. Hong, N. Yokoya, and W. He, “Guided deep decoder: Unsupervised image pair fusion,” in *Proceedings of the European Conference on Computer Vision (ECCV)*, pp. 87–102, 2020.

[21] L. Zhang, J. Nie, W. Wei, Y. Li, and Y. Zhang, “Deep blind hyperspectral image super-resolution,” *IEEE Transactions on Neural Networks and Learning Systems*, vol. 32, no. 6, pp. 2388–2400, 2020.

[22] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in *Advances in Neural Information Processing Systems (NeurIPS)*, pp. 2672–2680, 2014.

[23] N. Keshava and J. F. Mustard, “Spectral unmixing,” *IEEE Signal Processing Magazine*, vol. 19, no. 1, pp. 44–57, 2002.

[24] L. Loncan, L. B. De Almeida, J. M. Bioucas-Dias, X. Briottet, J. Chanussot, N. Dobigeon, S. Fabre, W. Liao, G. A. Licciardi, M. Simoes, et al., “Hyperspectral pansharpening: A review,” *IEEE Geoscience and Remote Sensing Magazine*, vol. 3, no. 3, pp. 27–46, 2015.

[25] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” *IEEE Transactions on Geoscience and Remote Sensing*, vol. 50, no. 2, pp. 528–537, 2011.

[26] N. Akhtar, F. Shafait, and A. Mian, “Sparse spatio-spectral representation for hyperspectral image super-resolution,” in *Proceedings of the European Conference on Computer Vision (ECCV)*, pp. 63–78, 2014.
[27] C. Lanaras, E. Baltsavias, and K. Schindler, "Hyperspectral super-resolution by coupled spectral unmixing," in *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, pp. 3586–3594, 2015.

[28] W. Dong, F. Fu, G. Shi, X. Cao, J. Wu, G. Li, and X. Li, "Hyperspectral image super-resolution via non-negative structured sparse representation," *IEEE Transactions on Image Processing*, vol. 25, no. 5, pp. 2337–2352, 2016.

[29] N. Akhtar, F. Shafait, and A. Mian, "Bayesian sparse representation for hyperspectral image super resolution," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 3631–3640, 2015.

[30] N. Akhtar, F. Shafait, and A. Mian, "Hierarchical beta process with gaussian process prior for hyperspectral image super resolution," in *Proceedings of the European Conference on Computer Vision (ECCV)*, pp. 103–120, 2016.

[31] R. Dian, L. Fang, and S. Li, "Hyperspectral image super-resolution via non-local sparse tensor factorization," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 5344–5353, 2017.

[32] Z. Zhu, J. Hou, J. Chen, H. Zeng, and J. Zhou, "Hyperspectral image super-resolution via deep progressive zero-centric residual learning," *IEEE Transactions on Image Processing*, vol. 30, pp. 1423–1438, 2021.

[33] Y. Fu, T. Zhang, Y. Zheng, D. Zhang, and H. Huang, "Hyperspectral image super-resolution with optimized rgb guidance," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 11661–11670, 2019.

[34] T. Zhang, Y. Fu, L. Wang, and H. Huang, "Hyperspectral image reconstruction using deep external and internal learning," in *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, pp. 8559–8568, 2019.

[35] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, pp. 2223–2232, 2017.

[36] M.-Y. Liu, T. Breuel, and J. Kautz, "Unsupervised image-to-image translation networks," in *Advances in Neural Information Processing Systems (NeurIPS)*, pp. 700–708, 2017.

[37] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang, "Diverse image-to-image translation via disentangled representations," in *Proceedings of the European Conference on Computer Vision (ECCV)*, pp. 35–51, 2018.

[38] K. Zheng, L. Gao, W. Liao, D. Hong, B. Zhang, X. Cui, and J. Chanussot, "Coupled convolutional neural network with adaptive response function learning for unsupervised hyperspectral super resolution," *IEEE Transactions on Geoscience and Remote Sensing*, 2020.

[39] H. Zhao, O. Gallo, I. Frosio, and J. Kautz, "Loss functions for image restoration with neural networks," *IEEE Transactions on Computational Imaging*, vol. 3, no. 1, pp. 47–57, 2016.

[40] A. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," *arXiv preprint arXiv:1807.03748*, 2018.

[41] D. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv preprint arXiv:1412.6980*, 2014.

[42] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," *arXiv preprint arXiv:1711.05101*, 2017.

[43] K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 1026–1034, 2015.

[44] F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar, "Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum," *IEEE Transactions on Image Processing*, vol. 19, no. 9, pp. 2241–2253, 2010.

[45] L. Wald, T. Ranchin, and M. Mangolini, "Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images," *Photogrammetric Engineering and Remote Sensing*, vol. 63, no. 6, pp. 691–699, 1997.

[46] J. A. Barsi, K. Lee, G. Kvaran, B. L. Markham, and J. A. Pedelty, "The spectral response of the landsat-8 operational land imager," *Remote Sensing*, vol. 6, no. 10, pp. 10232–10251, 2014.

[47] F. A. Kruse, A. Lefkoff, J. Boardman, K. Heidebrecht, A. Shapiro, P. Barloon, and A. Goetz, "The spectral image processing system (sips)-interactive visualization and analysis of imaging spectrometer data," *Remote Sensing of Environment*, vol. 44, no. 2-3, pp. 145–163, 1993.

[48] L. Wald, "Quality of high resolution synthesised images: Is there a simple criterion?," in *Proceedings of Conference Fusion Earth Data*, 2000.

[49] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," *IEEE Transactions on Image Processing*, vol. 13, no. 4, pp. 600–612, 2004.

[50] Z. Wang and A. C. Bovik, "A universal image quality index," *IEEE Signal Processing Letters*, vol. 9, no. 3, pp. 81–84, 2002.