<figure><img src="image_1.png" /></figure>

HAL
open science

# Graph-Induced Aligned Learning on Subspaces for Hyperspectral and Multispectral Data

Danfeng Hong, Naoto Yokoya, Jocelyn Chanussot, Jian Kang

► To cite this version:

Danfeng Hong, Naoto Yokoya, Jocelyn Chanussot, Jian Kang. Graph-Induced Aligned Learning on Subspaces for Hyperspectral and Multispectral Data. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (5), pp.4407-4418. 10.1109/TGRS.2020.3021140. hal-03429646

HAL Id: hal-03429646

https://hal.science/hal-03429646v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Graph-Induced Aligned Learning on Subspaces for Hyperspectral and Multispectral Data

Danfeng Hong<sup>1)</sup>, Member, IEEE, Jian Kang<sup>1)</sup>, Member, IEEE, Naoto Yokoya<sup>1)</sup>, Member, IEEE, and Jocelyn Chanussot<sup>1)</sup>, Fellow, IEEE

**Abstract**— In this article, we have great interest in investigating a common but practical issue in remote sensing (RS)—can a limited amount of one information-rich (or high-quality) data, e.g., hyperspectral (HS) image, improve the performance of a classification task using a large amount of another information-poor (low-quality) data, e.g., multispectral (MS) image? This question leads to a typical cross-modality feature learning. However, classic cross-modality representation learning approaches, e.g., manifold alignment, remain limited in effectively and efficiently handling such problems that the data from high-quality modality are largely absent. For this reason, we propose a novel graph-induced aligned learning (GiAL) framework by 1) adaptively learning a unified graph (further yielding a Laplacian matrix) from the data in order to align multimodality data (MS-HS data) into a latent shared subspace; 2) simultaneously modeling two regression behaviors with respect to labels and pseudo-labels under a multitask learning paradigm; and 3) dramatically updating the pseudo-labels according to the learned graph and refeeding the latest pseudo-labels into model learning of the next round. In addition, an optimization framework based on the alternating direction method of multipliers (ADMMs) is devised to solve the proposed GiAL model. Extensive experiments are conducted on two MS-HS RS data sets, demonstrating the superiority of the proposed GiAL compared with several state-of-the-art methods.

**Index Terms**— Cross-modality, fusion, graph learning, hyperspectral (HS), multispectral (MS), pseudo-labels, remote sensing (RS), semisupervised, subspace learning.

## I. INTRODUCTION

WITH the advent of the big data era and the rapid development of spaceborne or airborne optical imaging techniques, such as hyperspectral (HS) sensor, multispectral (MS) sensor, it is now easier to obtain a large amount of data than ever before. Unfortunately, most of them are

Manuscript received June 23, 2020; revised July 30, 2020; accepted August 30, 2020. This work was supported in part by the Japan Society for the Promotion of Science (JSPS) under Grant KAKENHI 18K18067 and in part by the AXA Research Fund. (Corresponding author: Jian Kang.)

Danfeng Hong is with the Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, 38000 Grenoble, France (e-mail: hongdanfeng1989@gmail.com).

Jian Kang is with the School of Electronic and Information Engineering, Soochow University, Suzhou 215006, China (e-mail: kangjian_1991@outlook.com).

Naoto Yokoya is with the Graduate School of Frontier Sciences, The University of Tokyo, Chiba 277-8561, Japan, and also with Geoinformatics Unit, RIKEN Center for Advanced Intelligence Project (AIP), RIKEN, Tokyo 03-0027, Japan (e-mail: naoto.yokoya@riken.jp).

Jocelyn Chanussot is with the Univ. Grenoble Alpes, INRIA, CNRS, Grenoble INP, LJK, 38000 Grenoble, France, and also with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China (e-mail: jocelyn@hi.is).

Color versions of one or more of the figures in this article are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TGRS.2020.3021140

usually unlabeled and of low-quality [1], especially for remote sensing (RS) data that are hardly labeled on a large scale due to their wide coverage in reality. Roughly, a large number of labeled samples are needed when training a reliable classifier, since such samples are able to capture the distribution of different categories more effectively, aiming at finding a more accurate decision boundary. Although high-quality training samples are expensive to collect, yet a small-size high-quality data may be available. For example, in [2], the multiscale superpixel shows powerful performance in integrating spatial and spectral information, and the proposed classifier can obtain high classification accuracy when it faces the problem of limited training samples and nonlinear features. As a result, the collection of HS images might remain limited, but the MS data, e.g., Sentinel-2, Landsat-8, are freely available on a large scale and even global scale. This naturally drives us to explore whether a limited amount of high-quality data is conducive to more discriminative feature learning on a large amount of low-quality data?

The question we raised above, which is related to cross-modality feature learning (CMFL) [3], is a special case of transfer learning (TL) [4] in the computer vision community but it widely exists in RS community [5]. A feasible solution to the problem is to transfer the richer and more discriminative information from the high-quality data, e.g., HS images [6], to the low-quality ones, e.g., MS images [7], or to jointly learn a shared subspace from the limited amount of high-quality and low-quality data correspondences. Recently, promising work around this topic has been widely applied in various tasks, e.g., text or image classification/recognition [8]–[14], data fusion [15]–[19], cross-modality data analysis [20]–[23], target detection [24]–[26], and many others. These approaches can be roughly categorized into shared subspace-based transfer learning (SSTL) and transductive transfer learning (TTL). We will resumptively analyze and discuss the connectivity between these existing methods and the CMFL task in the following.

Manifold alignment (MA) is one of the most popular feature-based alignment and representation learning frameworks that facilitate SSTL [8], [9], [27]–[30]. Wang and Mahadevan [8] proposed a semisupervised MA (SSMA) framework to learn a shared feature subspace by simultaneously considering the topological (graph) structures of labeled and unlabeled data. Tuia et al. [28] made the SSMA applicable to RS data alignment, thereby improving the classification performance effectively. Nevertheless, the SSMA’s performance is still limited by the low feature representative-ability.
The reasons for this are unfolded as follows. On the one hand, the graph structure of unlabeled data is directly constructed by similarity measurement, e.g., cosine or a radial basis function (RBF). On the other hand, there is no an effective connection between the shared subspace and label information. The transfer component analysis (TCA) presented by Pan et al. [9] is another representative work associated with SSTL. Different from the MA-based strategy, TCA starts from the perspective of statistics to find an aligned space where the source and target domains are expected to share the same data distributions and the topic labels. Herath et al. [29] proposed to learn a variant kernel subspace for domain adaption. A major bottleneck of TCA lies in selecting the kernel functions which are usually set manually, leading to a relatively weak transferability. Similar to TCA, joint learning (JL)-based methods aim at learning a latent shared subspace by measuring the reconstruction loss of original data and labels rather than the gap between data probabilistic distribution, which reveals a desirable performance particularly in cross-modal data retrieval [27] or action recognition [30]. However, most JL-based methods, to our knowledge, rarely consider the unlabeled information and intrinsic structure property of the data (e.g., data distribution).

According to the superiority of deep learning (DL) [31] techniques that are capable of capturing the nonlinear data distribution, to learn a joint feature representation over multimodalities, yielding a multimodal DL framework [32]. A two-stream autoencoder (AE) architecture (Corr-AE) was designed [20], which consists of two subnetworks (AEs). Each subnetwork is run independently to learn the discriminative features for each modality. In the process of network learning, the two subnetworks are enforced to share the same encoder features. Such a constraint ignores the correlation among samples. To this end, a correlational neural network (CorrNet) [33] was proposed by integrating AE and canonical correlation analysis (CCA). In CorrNet, the AE is utilized to learn high-level features for each modality while CCA is responsible for maximizing the correlation between the features of the two modalities. Although the DL-based methods tend to obtain the high-level or semantically meaningful features, yet a large amount of labeled samples is prerequisite.

Another mainstream approaches to tackle TL problems are based on TTL. Inspired by Zhu et al. [34] in which the graph-based label propagation (GLP) technique was designed for semisupervised learning, Long et al. [35] applied this technique to the TTL and proposed the adaptation regularization-based transfer learning (ARTL). More specifically, an undirected graph is first constructed by computing the similarities between samples of the different domains with a fixed kernel function (e.g., RBF) and the labels are then propagated to those unlabeled samples by means of the graph. The ARTL's performance basically relies on the selected kernel function and its parameter setting. To further improve the generalization ability of the ARTL, Chang et al. [36] conducted a generalized graph construction and completion by simultaneously taking the homogeneity and heterogeneity into account. In spite of this, its performance is inevitably hindered from going better, due to the lack of an adaptive graph

structure computation. Inspired by JL and MA techniques, Hong et al. [5] input two kinds of RS data, e.g., HS and MS data, and jointly estimated a common subspace under the guidance of label information, called common subspace learning (CoSpace). Beyond that, they further extended their model, yielding a semisupervised version-learnable manifold alignment (LeMA) [37], by fully considering the unlabeled samples and adaptively learning the graph structure not only between labeled samples but also between labeled samples and unlabeled ones.

To overcome the shortcomings of those methods mentioned above, we hold their essences back while get rid of their dross. To this end, we propose a graph-induced aligned learning framework, GiAL for short, to learn a common feature space from the MS-HS pairs by a data-driven graph learning. This not only makes the multimodal data comparable in a latent subspace and improves the explainability of the learned common subspace, further leading to a better transferability. More specifically, our contributions can be unfolded as follows.

1) A novel learning framework, called GiAL, is developed on MS-HS data by additionally introducing a regression task on unlabeled samples for cross-modality classification tasks.

2) Instead of directly computing the graph structure given by a fixed kernel function, GiAL adaptively learns a graph structure from the unlabeled samples itself and learns to align labeled samples and unlabeled ones in the form of a graph structure, yielding a better multimodal information blending.

3) Last but not least, we dynamically propagate the labels on the learned graph structure, yielding an iterateable updating pseudo-labels. The latest pseudo-labels can then be refed into model learning of the next round to further enhance the model's representation ability.

The remainder of this article is organized as follows. Section II presents the motivation briefly then details the methodology of our GiAL algorithm and the corresponding alternating direction method of multipliers (ADMM)-based optimization algorithm. In Section III, we show extensive experimental results and discussion on two popular MS-HS data sets both qualitatively and quantitatively. Finally, this article ends with a summary with the future outlook in Section IV.

## II. GIAL FRAMEWORK

In this section, we briefly state the cast problem of CMFL and our basic motivation. Accordingly, we formulate the methodology of the GiAL framework step by step and develop an ADMM-based [38]–[40] optimizer to solve the proposed model. Fig. 1 illustrates the holistic workflow of GiAL.

### A. Motivation

In many machine learning tasks such as classification and regression, data collection plays a key role, because high-quality training samples enable us to find a better decision boundary easily. However, in reality, high-quality training samples are costly to obtain; on the contrary, low-quality samples are relatively in large quantities and easy to be collected.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Illustration of the proposed GIAL in the MS-HS case.</figcaption></figure>

For instance, we can easily acquire MS satellite data at a larger scale from Sentinel-2 or Landsat-8 in land cover classification tasks. However, the corresponding HS satellite data can be collected only within tiny areas due to technical limitations of satellite sensors. This example motivates us to raise the question above related to CMFL.

## B. Method Overview

According to the aforementioned motivation, our goal in this article is to find a subspace or feature space, where the MS data are more representative and can be better classified. For this purpose, we propose a novel GIAL model by aligning the limited number of labeled HS and MS samples on the learnable graphs. In GIAL, multiple advanced learning-based techniques are embedded to learn better projection matrices linking to that ideal subspace.

More specifically, given the input of small-scale labeled HS and MS sample pairs, i.e., $X_L^1$ and $X_L^2$,

1. the shared subspace $E$ can be obtained by jointly learning two subspace projections ($\Theta_1$ and $\Theta_2$) from $X_L^1$ and $X_L^2$, respectively;

2. to project $X_L^1$ and $X_L^2$ into a shred subspace, e.g., $E$, we attempt to align the two subspaces ($\Theta_1 X_L^1$ and $\Theta_2 X_L^2$) by means of MA techniques;

3. to make the learned subspaces discriminative, we further bridge the to-be-estimated subspaces with the labels $Y_L$ by additionally learning a regression matrix $P$.

4. moreover, we also consider unlabeled MS data $X_U$, that is, simultaneously regressing labeled and unlabeled samples with labels $Y_L$ and updatable pseudo-labels $Y_{PL}$, yielding a multitask learning framework.

Fig. 2 shows a simplified flowchart of the proposed GIAL algorithm to outline the aforementioned content.

## C. Problem Formulation

Our studied problem is, in fact, a multimodal and multitask learning in CMFL. For that, we progressively push forward the modeling toward our GIAL framework. We start with a multi/cross-modal joint modeling to learn a latent shared subspace. Unlike the classic JL that connects only the learned subspace with labels, we simultaneously consider the labeled and unlabeled samples in a multitask regression fashion. Beyond the pseudo-labeling, we propose to dynamically update pseudo-labels by running GLP algorithm on a learnable graph.

The solutions to the above issue are, to our best knowledge, fourfold: 1) *JL*: a common latent subspace learning associated with labels is learned from multimodalities, in which the information is ideally shared and interacted; 2) *semisupervised learning*: unlabeled samples are fed into our GIAL learner
<table><caption>TABLE I
DEFINITION OF VARIABLES USED IN OUR GIAL ALGORITHM</caption><thead><tr><th>Variable</th><th>Definition</th><th>Type</th><th>Size</th></tr></thead><tbody><tr><td>d<sub>1</sub></td><td>the dimension of HS signatures</td><td>scalar</td><td>-</td></tr><tr><td>d<sub>2</sub></td><td>the dimension of MS signatures</td><td>scalar</td><td>-</td></tr><tr><td>c</td><td>the number of class</td><td>scalar</td><td>-</td></tr><tr><td>N</td><td>the number of labeled samples</td><td>scalar</td><td>-</td></tr><tr><td>M</td><td>the number of unlabeled samples</td><td>scalar</td><td>-</td></tr><tr><td>X&#770;<sub>L</sub><sup>1</sup></td><td>the labeled HS samples</td><td>matrix</td><td>d<sub>1</sub> × N</td></tr><tr><td>X&#770;<sub>L</sub><sup>2</sup></td><td>the labeled MS samples</td><td>matrix</td><td>d<sub>2</sub> × N</td></tr><tr><td>X&#770;<sub>L</sub></td><td>the labeled HS and MS samples</td><td>matrix</td><td>(d<sub>1</sub> + d<sub>2</sub>) × 2N</td></tr><tr><td>X<sub>U</sub></td><td>the unlabeled MS samples</td><td>matrix</td><td>d<sub>2</sub> × M</td></tr><tr><td>X&#770;<sub>PL</sub></td><td>the unlabeled samples, obtained by [0, X<sub>U</sub>]<sup>T</sup></td><td>matrix</td><td>(d<sub>1</sub> + d<sub>2</sub>) × M</td></tr><tr><td>Y<sub>L</sub></td><td>the one-hot encoding labels</td><td>matrix</td><td>c × N</td></tr><tr><td>Y<sub>PL</sub></td><td>the MS one-hot encoding pseudo-labels</td><td>matrix</td><td>c × M</td></tr><tr><td>Y&#770;<sub>PL</sub></td><td>the one-hot encoding pseudo-labels, obtained by [0, Y<sub>PL</sub>]</td><td>matrix</td><td>c × 2M</td></tr><tr><td>Y&#770;<sub>L</sub></td><td>the HS and MS one-hot encoding labels, obtained by [Y<sub>L</sub>, Y<sub>L</sub>]</td><td>matrix</td><td>c × 2N</td></tr><tr><td>Θ<sub>1</sub></td><td>the to-be-estimated subspace projection for HS data</td><td>matrix</td><td>d × d<sub>1</sub></td></tr><tr><td>Θ<sub>2</sub></td><td>the to-be-estimated HS-MS subspace projection for MS data</td><td>matrix</td><td>d × d<sub>2</sub></td></tr><tr><td>Θ</td><td>the to-be-estimated subspace projection, obtained by [Θ<sub>1</sub>, Θ<sub>2</sub>]</td><td>matrix</td><td>d × (d<sub>1</sub> + d<sub>2</sub>)</td></tr><tr><td>P</td><td>the linear regression matrix</td><td>matrix</td><td>c × d</td></tr><tr><td>W<sub>L</sub></td><td>the adjacency matrix of labeled samples</td><td>matrix</td><td>2N × 2N</td></tr><tr><td>D<sub>L</sub></td><td>the degree matrix of labeled samples</td><td>matrix</td><td>2N × 2N</td></tr><tr><td>L<sub>L</sub></td><td>the Laplacian matrix of labeled samples</td><td>matrix</td><td>2N × 2N</td></tr><tr><td>W&#770;</td><td>the joint adjacency matrix of labeled and unlabeled samples</td><td>matrix</td><td>(2N + M) × (2N + M)</td></tr><tr><td>L&#770;</td><td>the joint Laplacian matrix of labeled and unlabeled samples</td><td>matrix</td><td>(2N + M) × (2N + M)</td></tr><tr><td>Z</td><td>the pairwise Euclidean distance matrix</td><td>matrix</td><td>(2N + M) × (2N + M)</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Fig. 2. Flowchart of the proposed GIAL algorithm. In the (t)th iteration, we start with the input of labeled HS and MS samples as well as unlabeled MS samples and end up with the output of subspace projections Θ, regression matrix P, and the learned graph ˜W. Furthermore, the pseudo-labels in the (t+1)th can be updated by GLP on ˜W.</figcaption></figure>

to better capture the distribution of the original data thereby learn a more discriminative feature representation; 3) *data-driven graph learning*: the graph structure is learned from the data rather than computed based on given kernels; and 4) *pseudo-label iterative learning*: besides the labeled samples, unlabeled samples with pseudo-labels are equally involved into the model learning process for more discriminative feature representation learning. Given the definitions of variables used in our methodology, as detailed in Table I, we will gradually detail these techniques in the following.

1) *Multi/Cross-Modal Joint Learning*: Inspired by a popular latent subspace learning framework [41], we model the M/C-JL's problem by

$$ 
\begin{aligned}
& \min_{\mathbf{P},\mathbf{\Theta}} \frac{1}{2} \|\tilde{\mathbf{Y}}_L - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_L\|_{\mathrm{F}}^2 + \frac{\gamma}{2} \|\mathbf{P}\|_{\mathrm{F}}^2 + \frac{\beta}{2} \mathrm{tr}(\mathbf{E}\mathbf{L}_L\mathbf{E}^\mathrm{T}) \\
& \text{s.t. } \mathbf{E} = \mathbf{\Theta}\tilde{\mathbf{X}}_L, \quad \mathbf{\Theta}\mathbf{\Theta}^\mathrm{T} = \mathbf{I},
\end{aligned} 
\quad (1) 
$$

where $\tilde{\mathbf{X}}_L = \begin{bmatrix} \mathbf{X}_L^1 & \mathbf{0} \\ \mathbf{0} & \mathbf{X}_L^2 \end{bmatrix}$, $\gamma$ and $\beta$ are the regularization parameters to balance the different terms in (1). For previous feature extraction approaches, it is uncertain for the obtained features to have positive or negative effects on the subsequent classification task. Notably, the benefit of the JL model mainly lies in that the learned features, i.e., $\mathbf{\Theta}\tilde{\mathbf{X}}_L$, have great potential toward better classification results, owing to the use of the linear regression matrix P bridging the features with label information.

2) *Multitask Learning*: To effectively utilize the information of unlabeled samples, we also make a multitask regression not only between the labeled data and labels but also between the unlabeled samples and pseudo-labels

$$ \min_{\mathbf{P},\mathbf{\Theta}} \frac{\alpha}{2} \|\tilde{\mathbf{Y}}_L - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_L\|_{\mathrm{F}}^2 + \frac{\beta}{2} \mathrm{tr}(\mathbf{E}\mathbf{L}_L\mathbf{E}^\mathrm{T}) $$
$$ + \frac{1 - \alpha}{2} \|\tilde{\mathbf{Y}}_{PL} - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_{PL}\|_{F}^{2} + \frac{\gamma}{2} \|\mathbf{P}\|_{F}^{2} \\ \text{s.t. } \mathbf{E} = \mathbf{\Theta}[\tilde{\mathbf{X}}_{L}, \tilde{\mathbf{X}}_{PL}], \quad \mathbf{\Theta}\mathbf{\Theta}^{\mathrm{T}} = \mathbf{I} \quad (2) $$

where $\mathbf{I} \in \mathbb{R}^{d \times d}$ denotes the identify matrix and $\alpha$ is the regularization parameter. The pseudo-labels can be initialized by training a support vector machine (SVM) classifier on the training set.

### 3) Data-Driven Graph Learning
To further improve the learning ability of the multitask model, we learn a graph structure from the data and then propagate the labels on the learned graph, which aims at effectively optimizing the pseudo-label elements. However, most existing graph-based methods on TL (e.g., MA, TCA, and ARTL) are trapped in graph construction. The data-driven graph learning is conducted by adaptively learning a joint graph structure from the data instead of manually constructing the graph with the known kernels, then we have

$$ \begin{array}{ll} \min_{\mathbf{P}, \mathbf{\Theta}, \tilde{\mathbf{L}}} & \frac{\alpha}{2} \|\tilde{\mathbf{Y}}_{L} - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_{L}\|_{F}^{2} + \frac{\beta}{2} \mathrm{tr}(\mathbf{E}\tilde{\mathbf{L}}\mathbf{E}^{\mathrm{T}}) \\ & + \frac{1 - \alpha}{2} \|\tilde{\mathbf{Y}}_{PL} - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_{PL}\|_{F}^{2} + \frac{\gamma}{2} \|\mathbf{P}\|_{F}^{2} \\ \text{s.t. } \mathbf{E} = \mathbf{\Theta} & [\tilde{\mathbf{X}}_{L}, \tilde{\mathbf{X}}_{PL}], \quad \mathbf{\Theta}\mathbf{\Theta}^{\mathrm{T}} = \mathbf{I}, \quad \tilde{\mathbf{L}} = \tilde{\mathbf{L}}^{\mathrm{T}} \\ & \tilde{\mathbf{L}}_{i,j,i \neq j} \leq 0, \quad \tilde{\mathbf{L}}_{i,j,i=j} \geq 0, \quad \mathrm{tr}(\tilde{\mathbf{L}}) = s \quad (3) \end{array} $$

where the scalar $s > 0$ controls the scale.

To solve (3) effectively, we rewrite the trace term as

$$ \mathrm{tr}(\mathbf{E}\tilde{\mathbf{L}}\mathbf{E}^{\mathrm{T}}) = \frac{1}{2} \mathrm{tr}(\tilde{\mathbf{W}}\tilde{\mathbf{Z}}) = \frac{1}{2} \|\tilde{\mathbf{W}} \odot \mathbf{Z}\|_{1,1} \quad (4) $$

where $\mathbf{Z}$ is defined as a pairwise Euclidean distance matrix: $\mathbf{Z}_{i,j} = \|\mathbf{E}_i - \mathbf{E}_j\|^2$, and $\odot$ denotes the Schur-Hadamard (termwise) product.

Using (4), the optimization problem of smooth manifold in (3) can be equivalently converted to that of a graph sparsity as

$$ \begin{array}{ll} \min_{\mathbf{P}, \mathbf{\Theta}, \tilde{\mathbf{W}}} & \frac{\alpha}{2} \|\tilde{\mathbf{Y}}_L - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_L\|_F^2 + \frac{\beta}{4} \|\tilde{\mathbf{W}} \odot \mathbf{Z}\|_{1,1} \\ & + \frac{1-\alpha}{2} \|\tilde{\mathbf{Y}}_{PL} - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_{PL}\|_F^2 + \frac{\gamma}{2} \|\mathbf{P}\|_F^2 \\ \text{s.t. } \mathbf{E} = \mathbf{\Theta} & [\tilde{\mathbf{X}}_L, \tilde{\mathbf{X}}_{PL}], \quad \mathbf{\Theta}\mathbf{\Theta}^{\mathrm{T}} = \mathbf{I} \\ & \tilde{\mathbf{W}} = \tilde{\mathbf{W}}^{\mathrm{T}}, \quad \tilde{\mathbf{W}}_{i,j} \ge 0, \quad \|\tilde{\mathbf{W}}\|_{1,1} = s \quad (5) \end{array} $$

where $\|\tilde{\mathbf{W}} \odot \mathbf{Z}\|_{1,1}$ can be seen as a weighted sparsity problem, denoted as weighted $\ell_1$-norm of $\tilde{\mathbf{W}}$.

### 4) Our GiAL
With the learned graph structure [solving (5)] in hand, we can dynamically update the pseudo-labels ($\tilde{\mathbf{Y}}_{PL}$) by aids of the GLP in each big loop. The workflow continuously run until the pseudo-labels are not changed any more. This finally yields our GiAL framework.

## D. Model Learning Process

In the process of model learning, an iterative alternating optimization strategy is adopted by splitting the nonconvex problem (5) into a collection of convex subproblems with respect to the variables $\mathbf{P}$, $\mathbf{\Theta}$, and $\tilde{\mathbf{W}}$ as well as $\tilde{\mathbf{Y}}_{PL}$. An implementation of GiAL is summarized in Algorithm 1.

### Algorithm 1 GiAL

**Input:** $\tilde{\mathbf{Y}}_L$, $\tilde{\mathbf{X}}_L$, $\tilde{\mathbf{X}}_{PL}$, $\tilde{\mathbf{L}}$, $\alpha$, $\beta$, $\gamma$, $\mathrm{maxIter}$.

**Output:** $\mathbf{P}$, $\mathbf{\Theta}$, $\tilde{\mathbf{L}}$, $\tilde{\mathbf{Y}}_{PL}$

1. $t = 1$, $\zeta = 1 - 4$, $\mathbf{Obj} = 1 + \zeta$;

2. **Initializing $\mathbf{P}^t$, $\mathbf{\Theta}^t$, and $\tilde{\mathbf{Y}}_{PL}^t$**

3. **while** $\mathbf{Obj} \ge \zeta$ or $t \le \mathrm{maxIter}$ **do**

4. $k = 1$, $\mathbf{ObjIn} = 1 + \zeta$;

5. **while** $\mathbf{ObjIn} \ge \zeta$ or $i \le \mathrm{maxIter}$ **do**

6. Fix others to update $\mathbf{P}^k \succ \mathrm{Learning Regression}$

**Coefficients**

7. Fix others to update $\mathbf{\Theta}^k \succ \mathrm{Learning Subspace}$

**Projections**

8. Fix others to wisely update $\tilde{\mathbf{W}}^k$ instead of $\tilde{\mathbf{L}}^k$:

9. 1. compute $\mathbf{W}_L^k \succ \mathrm{Label-based Aligning}$

10. 2. update $\mathbf{W}^k$ and $\mathbf{W}^k \succ \mathrm{Learning to Align}$

11. 3. update $\mathbf{W}_U^k \succ \mathrm{Graph Learning}$

12. 4. collect $\mathbf{W}_L^k$, $\mathbf{W}^k$, $\mathbf{W}^k$, and $\mathbf{W}_U^k$ to be $\tilde{\mathbf{W}}^k$

13. 5. compute $\tilde{\mathbf{L}}^k = \tilde{\mathbf{D}}^k - \tilde{\mathbf{W}}^k$, $\tilde{\mathbf{D}}_{ii}^k = \sum_{i \neq j} \tilde{\mathbf{W}}_{ij}^k$ \\ $k \leftarrow k + 1$;

14. **end**

15. Update $\tilde{\mathbf{Y}}_{PL}^t$ with $\tilde{\mathbf{W}}^t \succ \mathrm{Updating Pseudo-labels}$

16. **end**

1) **Learning Regression Coefficients ($\mathbf{P}$):** This is a typical least-squares regression with Tikhonov regularization, which is formulated as follows:

$$ \min_{\mathbf{P}} \frac{\alpha}{2} \|\tilde{\mathbf{Y}}_L - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_L\|_F^2 + \frac{\gamma}{2} \|\mathbf{P}\|_F^2 \\ + \frac{1-\alpha}{2} \|\tilde{\mathbf{Y}}_{PL} - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_{PL}\|_F^2. \quad (6) $$

A closed-form solution of (6) is given by

$$ \mathbf{P} = (\alpha \tilde{\mathbf{Y}}_L \mathbf{H}_L + (1-\alpha) \tilde{\mathbf{Y}}_{PL} \mathbf{H}_{PL}) \times (\alpha \mathbf{H}_L \mathbf{H}_L^T + (1-\alpha) \mathbf{H}_{PL} \mathbf{H}_{PL}^T + \gamma \mathbf{I})^{-1} \quad (7) $$

where $\mathbf{H}_L = \mathbf{\Theta}\mathbf{X}_L$ and $\mathbf{H}_{PL} = \mathbf{\Theta}\mathbf{X}_{PL}$.

2) **Learning Subspace Projections ($\mathbf{\Theta}$):** The variable $\mathbf{\Theta}$ can be estimated by solving the following optimization problem:

$$ \begin{array}{ll} \min_{\mathbf{\Theta}} & \frac{\alpha}{2} \|\tilde{\mathbf{Y}}_L - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_L\|_F^2 + \frac{\beta}{2} \mathrm{tr}(\mathbf{E}\tilde{\mathbf{L}}\mathbf{E}^{\mathrm{T}}) \\ & + \frac{1-\alpha}{2} \|\tilde{\mathbf{Y}}_{PL} - \mathbf{P}\mathbf{\Theta}\tilde{\mathbf{X}}_{PL}\|_F^2 \\ \text{s.t. } \mathbf{E} = \mathbf{\Theta} & [\tilde{\mathbf{X}}_L, \tilde{\mathbf{X}}_{PL}], \quad \mathbf{\Theta}\mathbf{\Theta}^{\mathrm{T}} = \mathbf{I}. \quad (8) \end{array} $$

We convert (8) to an equivalent form by introducing auxiliary variables $\mathbf{J}$, $\mathbf{K}$, and $\mathbf{G}$ to replace $\mathbf{\Theta}\tilde{\mathbf{X}}_L$, $\mathbf{\Theta}\tilde{\mathbf{X}}_{PL}$, and $\mathbf{\Theta}$, respectively,

$$ \begin{array}{ll} \min_{\mathbf{\Theta}, \mathbf{J}, \mathbf{K}, \mathbf{G}} & \frac{\alpha}{2} \|\tilde{\mathbf{Y}}_L - \mathbf{P}\mathbf{J}\|_F^2 + \frac{1-\alpha}{2} \|\tilde{\mathbf{Y}}_{PL} - \mathbf{P}\mathbf{K}\|_F^2 \\ & + \frac{\beta}{2} \mathrm{tr}(\mathbf{E}\tilde{\mathbf{L}}\mathbf{E}^{\mathrm{T}}) \\ \text{s.t. } \mathbf{J} = \mathbf{\Theta}\tilde{\mathbf{X}}_L, \quad \mathbf{K} = \mathbf{\Theta}\tilde{\mathbf{X}}_{PL}, \quad \mathbf{G} = \mathbf{\Theta}, \quad \mathbf{GG}^{\mathrm{T}} = \mathbf{I}. \quad (9) \end{array} $$

The problem (9) can be solved by an ADMM-based optimization algorithm [42].
<figure><img src="image_5.png" /><figcaption>Fig. 3. Illustration of joint adjacency matrix $\tilde{W}$ that consists of six parts, i.e., $W_L$, $W_{LU}^1$, $W_{UL}^1$, $W_{LU}^2$, $W_{UL}^2$, and $W_U$.</figcaption></figure>

Please note that $\tilde{W}$ consists mainly of six parts, as shown in Fig. 3, they are $W_L$, $W_{LU}^1$, $W_{UL}^1$, $W_{LU}^2$, $W_{UL}^2$, and $W_U$, respectively. Due to the symmetry of $\tilde{W}$, we have $W_{LU}^1 = (W_{UL}^1)^T$ and $W_{LU}^2 = (W_{UL}^2)^T$. As a result, the optimization problem for the variable $\tilde{W}$ is degraded to four subproblems with respect to $W_L$, $W_{LU}^1$, $W_{LU}^2$, and $W_U$, respectively.

3) **Label-Based Aligning ($W_L$):** Given the labels for the training set, we can align class-specific samples to form a linear discriminant analysis (LDA)-like graph [43]

$$W_{L(i,j)} = \begin{cases} \frac{1}{N_k}, & X_i \text{ and } X_j \text{ are the } k\text{th class} \\ 0, & \text{otherwise.} \end{cases} \quad (10)$$

4) **Learning to Align ($W_{LU}^1$ or $W_{LU}^2$):** We propose to learn to align labeled and unlabeled samples in a common latent subspace in order to make an explicit connection between labeled samples and unlabeled ones. This leads to an optimization problem formulated as

$$\min_{W_{LU}^{1(2)}} \frac{\beta}{4} \|\mathbf{W} \odot \mathbf{Z}\|_{1,1} \quad \text{s.t. } 1/N_k \ge \mathbf{W}_{i,j} \ge 0, \quad \|\mathbf{W}\|_{1,1} = s \quad (11)$$

where the constraint $\mathbf{W}_{i,j} \le 1/N_k$ is added for the purpose of sharing same unit level with $W_L$. Likewise, (11) can be solved by ADMM [44], [45].

5) **Graph Learning ($W_U$):** Apart from learning to align labeled space with unlabeled samples, we also learn a graph structure of unlabeled samples for itself alignment, resulting in the following optimization problem:

$$\min_{W_U} \frac{\beta}{4} \|\mathbf{W} \odot \mathbf{Z}\|_{1,1} \\ \text{s.t. } \mathbf{W} = \mathbf{W}^T, \quad 1/N_k \ge \mathbf{W}_{i,j} \ge 0, \quad \|\mathbf{W}\|_{1,1} = s \quad (12)$$

which can be effectively solved based on ADMM as well.

6) **Dynamic Updating of Pseudo-Labels ($\tilde{Y}_{PL}$):** Given the pseudo-labels ($\tilde{Y}_{PL}^{(t)}$) of the $t$th step, we can correspondingly obtain the joint graph structure in the $t$th step ($\tilde{W}^{(t)}$) applied to infer the pseudo-labels of next step ($\tilde{Y}_{PL}^{(t+1)}$) by GLP, and then the updated pseudo-labels are fed into a new-round of loop again, which can be expressed as

$$\dots \tilde{Y}_{PL}^{(t+1)} \leftarrow \tilde{W}^{(t)} \leftarrow \tilde{Y}_{PL}^{(t)} \dots \quad (13)$$

The alternating strategy used in **Algorithm 1** is a block coordinate descent (BCD), which has been theoretically supported to converge to a stationary point as long as each subproblem in (5) is exactly minimized. The computational cost in the GiAL is dominated by matrix products, where the most costly step lies in solving $\Theta$, hence yielding an overall $O((d_1+d_2)(2N+M)^2)$ computational cost for (5).

## E. Discussion on Spatial and Spectral Resolution of HS and MS Images

To achieve better classification results on large-scale MS images, there are two typical solutions in the following:

1) **Image-Level:** Generating high-spectral resolution MS product by fusing low-spatial HS images and high-spatial MS images;

2) **Feature-Level:** Performing feature learning on HS and MS images to directly generate more discriminative feature representations.

Although enormous efforts have been made in the image-level fusion, e.g., [46], [47], yet its main limitation lies in the need of the identical geospatial location between HS and MS images. In reality, the HS images usually fail to acquire on a large scale, due to the limitations of imaging devices. This would lead to the difficulty in practical applications.

To this end, the feature-level fusion might be a good solution, particularly for the CMFL case, that is, the model is trained on the limited number of labeled HS and MS samples, while the HS data are absent in the testing phase. This meets the real situation of HS and MS data well. As a result, this article aims at investigating the feature-level fusion of HS and MS images in the CMFL case with a focus on the design of algorithms or models. For simplicity, we explore the HS and MS images with the same spatial resolution to verify the effectiveness of the proposed approach.

## III. EXPERIMENTS

In this section, pixel-wise cross-modality classification is explored as a potential application for quantitatively and qualitatively assessing the effectiveness of GiAL.

### A. Data Description and Preparation

The experiments are conducted on two MS-HS data sets.

1) **Houston MS-HS Data Sets:** The HS image was provided for the 2013 IEEE GRSS data fusion contest with 144 spectral bands, which was acquired by the ITRES-CASI sensor. It consists of 349 × 1905 pixels. Spectral simulation is performed to generate the MS image by degrading the HS image in the spectral domain using the MS spectral response functions (SRFs) of Sentinel-2 as filters (refer to [5] for more details). Using it, the MS data with dimensions of 349 × 1905 × 10 is generated. A MS image and a partial HS image as well as training and test labels are shown in Fig. 4(a). Table II lists the specific number of training and test samples.
<figure><img src="image_6.png" /><figcaption>Fig. 4. Exemplary data sets: (a) and (b) show the MS and the corresponding partial HS image as well as training and testing labels for the Houston and Chikusei MS-HS data sets, respectively.</figcaption></figure>

<table><caption>TABLE II
SCENE CATEGORY AND THE NUMBER OF TRAINING AND TEST SAMPLES USED IN THE HOUSTON MS-HS DATA SETS</caption><thead><tr><th>Class No.</th><th>Class Name</th><th>Training</th><th>Test</th></tr></thead><tbody><tr><td>1</td><td>Healthy Grass</td><td>537</td><td>699</td></tr><tr><td>2</td><td>Stressed Grass</td><td>61</td><td>1154</td></tr><tr><td>3</td><td>Synthetic Grass</td><td>340</td><td>357</td></tr><tr><td>4</td><td>Tree</td><td>209</td><td>1035</td></tr><tr><td>5</td><td>Soil</td><td>74</td><td>1168</td></tr><tr><td>6</td><td>Water</td><td>22</td><td>303</td></tr><tr><td>7</td><td>Residential</td><td>52</td><td>1203</td></tr><tr><td>8</td><td>Commercial</td><td>320</td><td>924</td></tr><tr><td>9</td><td>Road</td><td>76</td><td>1149</td></tr><tr><td>10</td><td>Highway</td><td>279</td><td>948</td></tr><tr><td>11</td><td>Railway</td><td>33</td><td>1185</td></tr><tr><td>12</td><td>Parking Lot1</td><td>329</td><td>904</td></tr><tr><td>13</td><td>Parking Lot2</td><td>20</td><td>449</td></tr><tr><td>14</td><td>Tennis Court</td><td>266</td><td>162</td></tr><tr><td>15</td><td>Running Track</td><td>279</td><td>381</td></tr><tr><td></td><td>Total</td><td>2897</td><td>12021</td></tr></tbody></table>

2) *Chikusei MS-HS Data Sets:* Similar to the Houston data, the MS data with dimensions of 2517 × 2335 × 10 was simulated from the corresponding HS data acquired by the Headwall's Hyperspec-VNIR-C sensor with dimensions of 2517 × 2335 × 128 [48]. The Fig. 4(b) shows the corresponding MS and partial HS images as well as training and test information, and the number of training and test sets are also detailed in Table III.

## B. Experimental Setup

1) *Evaluation Metrics:* We adopt three criteria, Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (κ), to quantify the experimental results using linear SVM (LSVM) classifier. For the LSVM, the well-known libsvm toolbox<fnref n="1" /> is used for implementation in our case. The considered LSVM uses the linearized kernel, whose optimal regularization hyperparameter C can be determined by fivefold

<table><caption>TABLE III
SCENE CATEGORY AND THE NUMBER OF TRAINING AND TEST SAMPLES USED IN THE CHIKUSEI MS-HS DATA SETS</caption><thead><tr><th>Class No.</th><th>Class Name</th><th>Training</th><th>Test</th></tr></thead><tbody><tr><td>1</td><td>Water</td><td>301</td><td>858</td></tr><tr><td>2</td><td>Bare Soil (School)</td><td>992</td><td>1867</td></tr><tr><td>3</td><td>Bare Soil (Farmland)</td><td>455</td><td>4397</td></tr><tr><td>4</td><td>Natural Plants</td><td>150</td><td>4272</td></tr><tr><td>5</td><td>Weeds in Farmland</td><td>928</td><td>1108</td></tr><tr><td>6</td><td>Forest</td><td>486</td><td>11904</td></tr><tr><td>7</td><td>Grass</td><td>989</td><td>5526</td></tr><tr><td>8</td><td>Rice Field (Grown)</td><td>813</td><td>8816</td></tr><tr><td>9</td><td>Rice Field (First Stage)</td><td>667</td><td>1268</td></tr><tr><td>10</td><td>Row Crops</td><td>377</td><td>5961</td></tr><tr><td>11</td><td>Plastic House</td><td>165</td><td>475</td></tr><tr><td>12</td><td>Manmade (Non-dark)</td><td>170</td><td>568</td></tr><tr><td>13</td><td>Manmade (Dark)</td><td>1291</td><td>6373</td></tr><tr><td>14</td><td>Manmade (Blue)</td><td>111</td><td>431</td></tr><tr><td>15</td><td>Manmade (Red)</td><td>35</td><td>187</td></tr><tr><td>16</td><td>Manmade Grass</td><td>21</td><td>1019</td></tr><tr><td>17</td><td>Asphalt</td><td>384</td><td>417</td></tr><tr><td></td><td>Total</td><td>8335</td><td>55447</td></tr></tbody></table>

cross validation in the range of $C = [10^{-2}, 10^{-1}, \dots, 10^{4}]$. More details can be found in [49]. It should be noted that the main reason for using the SVM classifier rather than other complex and advanced classifiers is to emphasize the performance gain derived from the learned shared features (or common subspaces).

2) *Comparison With State-of-the-Art Baselines:* Several state-of-the-art baselines related to our task (CMFL) are selected for comparison, as briefly described below.

1) *O-Baseline:* We directly train a classifier on target-domain using original image features.

2) *Supervised Manifold Alignment (SMA):* The different modalities are aligned by a label-driven MA to find a latent low-dimensional space where the information can be effectively shared. The code is available from the website.<fnref n="2" />

<footnote n="1">¹https://www.csie.ntu.edu.tw/~cjlin/libsvm/</footnote>

<footnote n="2">²https://sites.google.com/site/changwangnk/home/ma-html</footnote>
<table><caption>TABLE IV
QUANTITATIVE PERFORMANCE COMPARISON OF DIFFERENT ALGORITHMS ON THE HOUSTON MS-HS DATA SETS. THE BEST ONE IS SHOWN IN BOLD</caption><thead><tr><th>Methods</th><th>O-Baseline</th><th>SMA</th><th>SSMA</th><th>ARTL</th><th>MDL</th><th>Corr-AE</th><th>CoSpace</th><th>LeMA</th><th>GiAL</th></tr></thead><tbody><tr><td>OA (%)</td><td>62.12</td><td>68.01</td><td>69.29</td><td>68.51</td><td>72.63</td><td>71.34</td><td>69.38</td><td>73.42</td><td><strong>80.66</strong></td></tr><tr><td>AA (%)</td><td>65.97</td><td>70.50</td><td>72.00</td><td>71.06</td><td>74.12</td><td>73.27</td><td>71.69</td><td>74.76</td><td><strong>81.31</strong></td></tr><tr><td>κ</td><td>0.5889</td><td>0.6520</td><td>0.6659</td><td>0.6578</td><td>0.7027</td><td>0.6884</td><td>0.6672</td><td>0.7110</td><td><strong>0.7896</strong></td></tr><tr><td>Class 1</td><td>76.39</td><td>75.25</td><td>74.25</td><td>87.84</td><td>70.96</td><td>85.69</td><td>75.54</td><td><strong>89.56</strong></td><td>87.70</td></tr><tr><td>Class 2</td><td>80.59</td><td>97.57</td><td>97.57</td><td>76.86</td><td>86.74</td><td>81.02</td><td>73.74</td><td>93.67</td><td><strong>98.44</strong></td></tr><tr><td>Class 3</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td></tr><tr><td>Class 4</td><td>85.51</td><td>94.78</td><td>95.85</td><td>96.71</td><td>98.36</td><td>97.00</td><td><strong>98.74</strong></td><td>97.49</td><td>97.20</td></tr><tr><td>Class 5</td><td>99.06</td><td>98.97</td><td>99.32</td><td>99.57</td><td>99.49</td><td>98.54</td><td>99.40</td><td>99.49</td><td><strong>100.00</strong></td></tr><tr><td>Class 6</td><td>86.14</td><td>86.47</td><td><strong>99.67</strong></td><td>85.81</td><td>86.14</td><td>84.82</td><td>85.48</td><td>86.47</td><td>99.34</td></tr><tr><td>Class 7</td><td>50.62</td><td>72.32</td><td>72.15</td><td>63.67</td><td>70.82</td><td>72.15</td><td>73.98</td><td><strong>83.21</strong></td><td>81.80</td></tr><tr><td>Class 8</td><td>56.49</td><td>62.01</td><td><strong>64.61</strong></td><td>55.19</td><td>64.07</td><td>55.52</td><td>63.53</td><td>62.77</td><td>60.93</td></tr><tr><td>Class 9</td><td>56.22</td><td>49.96</td><td>50.57</td><td>62.40</td><td>66.93</td><td>65.27</td><td>59.79</td><td>64.49</td><td><strong>67.89</strong></td></tr><tr><td>Class 10</td><td>45.36</td><td>58.12</td><td>58.33</td><td>45.15</td><td>57.38</td><td><strong>69.94</strong></td><td>64.14</td><td>60.97</td><td>65.51</td></tr><tr><td>Class 11</td><td>27.43</td><td>28.86</td><td>36.46</td><td>47.34</td><td>47.17</td><td>41.10</td><td>36.54</td><td>41.27</td><td><strong>69.20</strong></td></tr><tr><td>Class 12</td><td>31.64</td><td>35.84</td><td>34.18</td><td>48.23</td><td>63.61</td><td>50.66</td><td>46.79</td><td>45.02</td><td><strong>86.06</strong></td></tr><tr><td>Class 13</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.67</td><td>2.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td><strong>6.46</strong></td></tr><tr><td>Class 14</td><td>97.53</td><td><strong>100.00</strong></td><td>99.38</td><td>98.77</td><td><strong>100.00</strong></td><td>99.38</td><td><strong>100.00</strong></td><td>99.38</td><td><strong>100.00</strong></td></tr><tr><td>Class 15</td><td>96.59</td><td>97.38</td><td>97.64</td><td>97.64</td><td>98.16</td><td>97.90</td><td>97.64</td><td>97.64</td><td><strong>99.21</strong></td></tr></tbody></table>

<table><caption>TABLE V
QUANTITATIVE PERFORMANCE COMPARISON OF DIFFERENT ALGORITHMS ON THE CHIKUSEI MS-HS DATA SETS. THE BEST ONE IS SHOWN IN BOLD</caption><thead><tr><th>Methods</th><th>O-Baseline</th><th>SMA</th><th>SSMA</th><th>ARTL</th><th>MDL</th><th>Corr-AE</th><th>CoSpace</th><th>LeMA</th><th>GiAL</th></tr></thead><tbody><tr><td>OA (%)</td><td>60.20</td><td>67.90</td><td>69.68</td><td>69.92</td><td>72.10</td><td>71.65</td><td>71.12</td><td>75.11</td><td><strong>80.42</strong></td></tr><tr><td>AA (%)</td><td>69.42</td><td>70.79</td><td>72.27</td><td>73.45</td><td>73.95</td><td>67.42</td><td>73.96</td><td>75.29</td><td><strong>79.68</strong></td></tr><tr><td>κ</td><td>0.5523</td><td>0.6391</td><td>0.6602</td><td>0.6632</td><td>0.6871</td><td>0.6819</td><td>0.6746</td><td>0.7194</td><td><strong>0.7800</strong></td></tr><tr><td>Class 1</td><td>78.21</td><td>98.72</td><td>99.53</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>99.30</td><td>92.54</td><td>98.25</td><td>98.83</td></tr><tr><td>Class 2</td><td>94.43</td><td>93.20</td><td>93.20</td><td>80.45</td><td>80.50</td><td>92.93</td><td>93.47</td><td>93.20</td><td><strong>94.59</strong></td></tr><tr><td>Class 3</td><td>23.54</td><td>62.57</td><td>68.41</td><td>87.56</td><td>87.42</td><td>73.19</td><td>80.40</td><td>89.29</td><td><strong>89.36</strong></td></tr><tr><td>Class 4</td><td>92.13</td><td>90.57</td><td>92.51</td><td>87.73</td><td>87.64</td><td>88.55</td><td>90.59</td><td>95.11</td><td><strong>97.40</strong></td></tr><tr><td>Class 5</td><td><strong>97.65</strong></td><td>28.43</td><td>24.01</td><td>51.44</td><td>49.73</td><td>67.33</td><td>83.94</td><td>60.74</td><td>86.28</td></tr><tr><td>Class 6</td><td>62.01</td><td>62.52</td><td>68.27</td><td>65.32</td><td>73.77</td><td>75.24</td><td>63.61</td><td>76.34</td><td><strong>88.47</strong></td></tr><tr><td>Class 7</td><td><strong>99.67</strong></td><td>96.87</td><td>95.40</td><td>96.74</td><td>97.21</td><td>99.08</td><td>97.74</td><td>97.63</td><td>94.93</td></tr><tr><td>Class 8</td><td>57.11</td><td>95.59</td><td>96.88</td><td>89.88</td><td>92.38</td><td>96.43</td><td>95.05</td><td><strong>99.27</strong></td><td>96.29</td></tr><tr><td>Class 9</td><td><strong>100.00</strong></td><td>99.53</td><td>99.45</td><td>95.58</td><td>95.66</td><td>97.79</td><td>98.66</td><td>99.76</td><td><strong>100.00</strong></td></tr><tr><td>Class 10</td><td>24.81</td><td>21.39</td><td>20.94</td><td>25.42</td><td>25.26</td><td>23.70</td><td>22.35</td><td>26.47</td><td><strong>51.20</strong></td></tr><tr><td>Class 11</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.63</td><td><strong>65.26</strong></td></tr><tr><td>Class 12</td><td>90.32</td><td>90.14</td><td>90.14</td><td>90.14</td><td>90.14</td><td>87.50</td><td>90.32</td><td>88.38</td><td><strong>96.30</strong></td></tr><tr><td>Class 13</td><td><strong>33.11</strong></td><td>32.61</td><td>31.32</td><td><strong>33.11</strong></td><td><strong>33.11</strong></td><td>33.09</td><td><strong>33.11</strong></td><td><strong>33.11</strong></td><td><strong>33.11</strong></td></tr><tr><td>Class 14</td><td>94.20</td><td>72.85</td><td>94.20</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>5.57</td><td>59.40</td><td>45.01</td><td>68.91</td></tr><tr><td>Class 15</td><td><strong>100.00</strong></td><td>93.58</td><td><strong>100.00</strong></td><td>88.77</td><td>88.77</td><td>91.44</td><td>93.58</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td></tr><tr><td>Class 16</td><td>74.88</td><td>99.71</td><td><strong>99.80</strong></td><td>98.53</td><td>98.53</td><td>87.63</td><td>97.84</td><td>98.04</td><td>83.12</td></tr><tr><td>Class 17</td><td>58.03</td><td>65.23</td><td>62.11</td><td>58.03</td><td>57.07</td><td>27.34</td><td>64.75</td><td><strong>78.66</strong></td><td>10.55</td></tr></tbody></table>

3) **SSMA [8]:** An SSMA alignment is applied to learn low-dimensional cross-modal representations from labeled and unlabeled data. The code is available from the website.<fnref n="3" />

4) **ARTL [35]:** ARTL tries to construct a kernel on which the GLP technique is leveraged to infer the labels. The code is available from the website.<fnref n="4" />

5) **MDL [32]:** As a DL's pioneer to multimodal application, multimodal deep learning (MDL) learns a joint feature representation over the encoder layers generated by AEs for each modality.

6) **CorrNet [33]:** Similar to Corr-AE, AE is responsible for extracting features of each modality, while CCA

serves as a link with the features by maximizing their correlations. The code is available from the website.<fnref n="5" />

7) **CoSpace [5]:** CoSpace jointly learns a common subspace across the modalities the guidance of label information. The code is available from the website.<fnref n="6" />

8) **LeMA [37]:** Different from CoSpace, LeMA further considers the unlabeled samples by the means of data-driven graph learning instead of a fixed graph structure, yielding more discriminative feature learning and tending to find a better decision boundary. The code is available from the website.<fnref n="7" />

The parameter configuration for the algorithms plays a key role in performance assessment. Herein, the parameters of different methods are determined by a tenfold cross-validation

<footnote n="3">³https://sites.google.com/site/changwangnk/home/ma-html</footnote>

<footnote n="4">⁴https://github.com/jindongwang/transferlearning/tree/master/code</footnote>

<footnote n="5">⁵https://github.com/apsarath/CorrNet</footnote>

<footnote n="6">⁶https://sites.google.com/view/danfeng-hong/data-code</footnote>

<footnote n="7">⁷https://sites.google.com/view/danfeng-hong/data-code</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 5. Confusion matrix of GiAL on two MS-HS data sets.</figcaption></figure>

on the training set. This is a common tactic that has been widely used in model selection.

It should be emphasized, however, that this article aims at assessing the representation and generalization ability of features extracted by different algorithms in the CMFL (i.e., HS-MS) case. In other words, these competitors and our proposed GiAL are feature extraction methods, either supervised or semisupervised. Similar to some relevant works [5], [27], [28], [30], [37], [41], where the feature extraction step and classification are separately performed, in our GiAL, the features are first extracted using labeled or unlabeled information and then fed into a classifier, e.g., SVM, for further classification.

### C. Results and Analysis With MS-HS Data

Tables IV and V list the quantitative results of the compared algorithms on two different MS-HS data sets, while Figs. 6 and 7 visually highlight the classification maps of these compared methods on the Houston and Chikusei data sets, respectively.

Overall, O-Baseline yields a worst classification performance in terms of OA, AA, and $\kappa$ as well as the accuracies for each class by only using MS data. The most possible reason is lack of rich and discriminate information (e.g., HS data). For those MA-based approaches, the SMA obviously performs better than MS only with the multimodal input (MS-HS). This, to some extent, indicates that the limited amount of HS data can guide the corresponding MS data toward better discriminative feature representations. By introducing unlabeled samples into the MA framework, SSMA can help us better capture the real data distribution, thereby obtain more accurate decision boundaries. Similarly, ARTL shows the competitive results by inferring labels of the unlabeled samples on a given graph. However, the aforementioned algorithms, to a great extent, depends on the eigendecomposition, thus expensive computational cost and more freedoms hinder the performance being further improved.

The DL-based approaches (e.g., MDL, Corr-AE), which nonlinearly learn features in a more discriminative fashion, outperform the previous ones at an increase of about 3% OA, AA, and $\kappa$, owing to more powerful data fitting ability in extracting features. Nevertheless, their ability in enhancing the subspace representations remains limited, due to the lack of the direct label guidance in the process of model learning.

Inspired by joint subspace learning and regression technique, the supervised CoSpace and semisupervised LeMA achieve the competitive performance in terms of the classification accuracy on both MS-HS data sets, in which the LeMA obviously performs better due to the full consideration of unlabeled samples. Owing to the use of multimodalities, labels and pseudo-labels as well as data-driven graph learning, the performance of GiAL is much more superior to those of the other methods as can be observed in Tables IV and V. This demonstrates that GiAL is likely to learn a more discriminative feature representation and to find a better decision boundary. We also provide the confusion matrix obtained by GiAL on two MS-HS data sets to highlight the class-specific performances, as shown in Fig. 5.

Although the proposed GiAL does not always perform the optimal performance for each class compared to other competitors, the classification accuracies obtained by GiAL rank the first place for most of the classes and the second place for the rest on the Houston data sets. There is a similar trend on the Chikusei data sets as well. These have well demonstrated the superiority of the proposed method, since it is hardly difficult to find a method that can beat all the others in accuracies for each class in reality.

### D. Visual Comparison

Figs. 6 and 7 visualize the classification maps of several state-of-the-art compared methods and our GiAL. Intuitively, the classification results of the proposed method are more smooth and realistic on both data sets, which are approaching the real scenes (see Fig. 4) compared to other competitors. In particular, the regions covered by the cloud on the Houston data sets are recognized well, which is robust against the shadow effects. Moreover, the performance gain can be observed successively from Baseline, MA-based methods (e.g., SMA, SSMA, and ARTL), DL-based methods (e.g., MDL and Corr-AE), JL-based models (e.g., CoSpace and LeMA), and the newly-proposed GiAL. In the other Chikusei scene, one material: *Man-made-D* that is located on the bottom right corner, is identified effectively, unlike other compared methods that misclassify it to be *Bare Soil*. There is basically a consistent trend between qualitative (see Figs. 6 and 7) and quantitative (see Tables IV and V) results.

### E. Parameter Sensitivity Analysis

We investigate the sensitivity of the proposed method to three regularization parameters ($\alpha$, $\beta$, and $\gamma$) and the subspace dimension, as visualized in Fig. 8. The subspace dimension is a common parameter for all methods, which is screened from {5, 10, 15, ..., 100}. The balance parameter ($\alpha$) for two reconstruction terms with respect to labels and pseudo-labels is varied in the range of {0, 0.1, 0.2, ..., 1.0}, where $\alpha = 0$ means no labels considered (only pseudo-labels), otherwise no pseudo-labels considered (only labels) at $\alpha = 1$. $\beta$ and $\gamma$ control the performances of learning regression coefficients and graph structure, respectively. We vary them in the range of {10$^{-4}$, 10$^{-3}$, ..., 10$^{0}$, 10$^{1}$} by fixing other parameters. As can be observed from Fig. 8, when $\alpha$, $\beta$, $\gamma$, and the subspace
<figure><img src="image_8.png" /><figcaption>Fig. 6. Classification maps of different algorithms on the Houston MS-HS data sets.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Classification maps of different algorithms on the Chikusei MS-HS data sets.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 8. Parameter sensitivity analysis of three regularization (e.g., α, β, and γ) and the number of iterations of updating pseudo-labels in GiAL and the dimensionality of the subspace on the Houston data set.</figcaption></figure>

dimension are set to 0.6, 0.1, 0.1, and 30, respectively, GiAL reaches satisfactory performance.

Besides, the model's stability with the increase of the number of iterations in updating pseudo-labels is also investigated. As can be seen from Fig. 8, our dynamic updating strategy for pseudo-labels via GLP has shown to be effective and obtained an obvious performance improvement. More remarkably, our GiAL model can reach stable results when the pseudo-labels

are no longer changed with more iterations, as shown in the last subfigure of Fig. 8, where the OAs are fixed after around ten iterations.

## IV. CONCLUSION

In reality, a large amount of MS RS data can be collected easily and largely. Rather, high-quality HS RS data are expensive and difficult to obtain on a large scale. This motivates us
to investigate whether a limited amount of high-quality data can contribute to recognition or classification tasks with a large amount of low-quality data. For this reason, we propose GIAL framework related to the CMFL's issue, which jointly utilizes the labels and pseudo-labels, and dynamically updates the pseudo-labeled information by adaptively learning a Laplacian matrix on the latent subspaces rather than using a graph structure given by specified functions, e.g., RBF. Extensive experiments are conducted using different algorithms on two cross-modal MS-HS data sets. The results of GIAL demonstrates its superiority and effectiveness for MS-HS data. In the future work, we would like to enhance the representation ability in a nonlinear fashion, e.g., using DL techniques, and develop more advanced feature learning algorithms to address more challenging CMFL-related cases, e.g., HS and MS images with different spatial resolution.

## ACKNOWLEDGMENT

The authors would like to thank the Hyperspectral Image Analysis Group, University of Houston and the IEEE GRSS DFC2013 for providing the University of Houston HS data set.

## REFERENCES

[1] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, "An augmented linear mixing model to address spectral variability for hyperspectral unmixing," *IEEE Trans. Image Process.*, vol. 28, no. 4, pp. 1923-1938, Apr. 2019.

[2] H. Yu, L. Gao, W. Liao, B. Zhang, A. Pižurica, and W. Philips, "Multiscale superpixel-level subspace-based support vector machines for hyperspectral image classification," *IEEE Geosci. Remote Sens. Lett.*, vol. 14, no. 11, pp. 2142-2146, Nov. 2017.

[3] D. Hong, N. Yokoya, G.-S. Xia, J. Chanussot, and X. X. Zhu, "X-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing data," *ISPRS J. Photogramm. Remote Sens.*, vol. 167, pp. 12-23, Sep. 2020.

[4] S. J. Pan and Q. Yang, "A survey on transfer learning," *IEEE Trans. Knowl. Data Eng.*, vol. 22, no. 10, pp. 1345-1359, Oct. 2010.

[5] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, "CoSpace: Common subspace learning from hyperspectral-multispectral correspondences," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 7, pp. 4349-4359, Jul. 2019.

[6] M. Tang, L. Gao, A. Marinoni, P. Gamba, and B. Zhang, "Integrating spatial information in the normalized P-linear algorithm for nonlinear hyperspectral unmixing," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 11, no. 4, pp. 1179-1190, Apr. 2018.

[7] X. Wu, D. Hong, J. Chanussot, Y. Xu, R. Tao, and Y. Wang, "Fourier-based rotation-invariant feature boosting: An efficient framework for geospatial object detection," *IEEE Geosci. Remote Sens. Lett.*, vol. 17, no. 2, pp. 302-306, Feb. 2020.

[8] C. Wang and S. Mahadevan, "Heterogeneous domain adaptation using manifold alignment," in *Proc. IJCAI*, 2011, pp. 1541-1546.

[9] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, "Domain adaptation via transfer component analysis," *IEEE Trans. Neural Netw.*, vol. 22, no. 2, pp. 199-210, Feb. 2011.

[10] D. Hong, N. Yokoya, and X. X. Zhu, "Learning a robust local manifold representation for hyperspectral dimensionality reduction," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 10, no. 6, pp. 2960-2975, Jun. 2017.

[11] D. Hong, L. Gao, J. Yao, B. Zhang, P. Antonio, and J. Chanussot, "Graph convolutional networks for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, early access, Aug. 18, 2020, doi: 10.1109/TGRS.2020.3015157.

[12] R. Hang, Q. Liu, D. Hong, and P. Ghamisi, "Cascaded recurrent neural networks for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 8, pp. 5384-5394, Aug. 2019.

[13] B. Rasti et al., "Feature extraction for hyperspectral imagery: The evolution from shallow to deep (overview and toolbox)," *IEEE Geosci. Remote Sens. Mag.*, early access, Apr. 29, 2020, doi: 10.1109/MGRS.2020.2979764.

[14] D. Hong, X. Wu, P. Ghamisi, J. Chanussot, N. Yokoya, and X. X. Zhu, "Invariant attribute profiles: A spatial-frequency joint feature extractor for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 6, pp. 3791-3808, Jun. 2020.

[15] Q. Yang, "Cross-domain data fusion," *Computer*, vol. 49, no. 4, p. 18, 2016.

[16] Y. Xu, Z. Wu, J. Chanussot, P. Comon, and Z. Wei, "Nonlocal coupled tensor CP decomposition for hyperspectral and multispectral image fusion," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 1, pp. 348-362, Jan. 2020.

[17] X. Liu, C. Deng, J. Chanussot, D. Hong, and B. Zhao, "StfNet: A two-stream convolutional neural network for spatiotemporal image fusion," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 9, pp. 6552-6564, Sep. 2019.

[18] L. Gao, D. Hong, J. Yao, B. Zhang, P. Gamba, and J. Chanussot, "Spectral superresolution of multispectral imagery with joint sparse and low-rank learning," *IEEE Trans. Geosci. Remote Sens.*, early access, Jun. 18, 2020, doi: 10.1109/TGRS.2020.300684.

[19] J. Yao, D. Hong, J. Chanussot, D. Meng, X. Zhu, and Z. Xu, "Cross-attention in coupled unmixing nets for unsupervised hyperspectral super-resolution," 2020, arXiv:2007.05230. [Online]. Available: http://arxiv.org/abs/2007.05230

[20] F. Feng, X. Wang, and R. Li, "Cross-modal retrieval with correspondence autoencoder," in *Proc. ACM MM*, 2014, pp. 7-16.

[21] D. Tuia, D. Marcos, and G. Camps-Valls, "Multi-temporal and multi-source remote sensing image classification by nonlinear relative normalization," *ISPRS J. Photogramm. Remote Sens.*, vol. 120, pp. 1-12, Oct. 2016.

[22] J. Kang, M. Körner, Y. Wang, H. Taubenböck, and X. X. Zhu, "Building instance classification using street view images," *ISPRS J. Photogramm. Remote Sens.*, vol. 145, pp. 44-59, Nov. 2018.

[23] D. Hong et al., "More diverse means better: Multimodal deep learning meets remote-sensing imagery classification," *IEEE Trans. Geosci. Remote Sens.*, early access, Aug. 24, 2020, doi: 10.1109/TGRS.2020.3016820.

[24] L. Zhang, L. Zhang, D. Tao, and X. Huang, "Sparse transfer manifold embedding for hyperspectral target detection," *IEEE Trans. Geosci. Remote Sens.*, vol. 52, no. 2, pp. 1030-1043, Feb. 2014.

[25] C. Li, L. Gao, Y. Wu, B. Zhang, J. Plaza, and A. Plaza, "A real-time unsupervised background extraction-based target detection method for hyperspectral imagery," *J. Real-Time Image Process.*, vol. 15, no. 3, pp. 597-615, Oct. 2018.

[26] X. Wu, D. Hong, J. Tian, J. Chanussot, W. Li, and R. Tao, "ORSIm detector: A novel object detection framework in optical remote sensing imagery using spatial-frequency channel features," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 7, pp. 5146-5158, Jul. 2019.

[27] K. Wang, R. He, W. Wang, L. Wang, and T. Tan, "Learning coupled feature spaces for cross-modal matching," in *Proc. ICCV*, 2013, pp. 2088-2095.

[28] D. Tuia, M. Volpi, M. Trolliet, and G. Camps-Valls, "Semisupervised manifold alignment of multimodal remote sensing images," *IEEE Trans. Geosci. Remote Sens.*, vol. 52, no. 12, pp. 7708-7720, Dec. 2014.

[29] S. Herath, M. T. Harandi, and F. Porikli, "Learning an invariant Hilbert space for domain adaptation," in *Proc. CVPR*, 2017, pp. 3956-3965.

[30] J.-F. Hu, W.-S. Zheng, J. Lai, and J. Zhang, "Jointly learning heterogeneous features for RGB-D activity recognition," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 39, no. 11, pp. 2186-2200, Nov. 2017.

[31] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," *Nature*, vol. 521, no. 7553, p. 436, 2015.

[32] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep learning," in *Proc. ICML*, 2011, pp. 689-696.

[33] S. Chandar, M. M. Khapra, H. Larochelle, and B. Ravindran, "Correlational neural networks," *Neural Comput.*, vol. 28, no. 2, pp. 257-285, Feb. 2016.

[34] X. Zhu, Z. Ghahramani, and J. Lafferty, "Semi-supervised learning using Gaussian fields and harmonic functions," in *Proc. ICML*, 2003, pp. 912-919.

[35] M. Long, J. Wang, G. Ding, S. J. Pan, and P. S. Yu, "Adaptation regularization: A general framework for transfer learning," *IEEE Trans. Knowl. Data Eng.*, vol. 26, no. 5, pp. 1076-1089, May 2014.

[36] W. Chang, Y. Wu, H. Liu, and Y. Yang, "Cross-domain kernel induction for transfer learning," in *Proc. AAAI*, 2017, pp. 1763-1769.

[37] D. Hong, N. Yokoya, N. Ge, J. Chanussot, and X. X. Zhu, "Learnable manifold alignment (LeMA): A semi-supervised cross-modality learning framework for land cover and land use classification," *ISPRS J. Photogramm. Remote Sens.*, vol. 147, pp. 193-205, Jan. 2019.
[38] D. Hong and X. X. Zhu, “SULoRA: Subspace unmixing with low-rank attribute embedding for hyperspectral data analysis,” *IEEE J. Sel. Topics Signal Process.*, vol. 12, no. 6, pp. 1351–1363, Dec. 2018.

[39] Y. Huang, L. Zhang, J. Li, Z. Chen, and X. Yang, “Reweighted tensor factorization method for SAR narrowband and wideband interference mitigation using smoothing multiview tensor model,” *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 5, pp. 3298–3313, May 2020.

[40] J. Kang, Y. Wang, and X. X. Zhu, “Multipass sar interferometry based on total variation regularized robust low rank tensor decomposition,” *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 8, pp. 5354–5366, Aug. 2020.

[41] S. Ji and J. Ye, “Linear dimensionality reduction for multi-label classification,” in *Proc. IJCAI*, 2009, pp. 1077–1082.

[42] C. Sagonas, Y. Panagakis, A. Leidinger, and S. Zafeiriou, “Robust joint and individual variance explained,” in *Proc. CVPR*, vol. 2, 2017, p. 6.

[43] D. Hong, N. Yokoya, J. Chanussot, J. Xu, and X. X. Zhu, “Learning to propagate labels on graphs: An iterative multitask regression framework for semi-supervised hyperspectral dimensionality reduction,” *ISPRS J. Photogramm. Remote Sens.*, vol. 158, pp. 35–49, Dec. 2019.

[44] F. Heide, W. Heidrich, and G. Wetzstein, “Fast and flexible convolutional sparse coding,” in *Proc. CVPR*, 2015, pp. 5135–5143.

[45] J. Kang, D. Hong, J. Liu, G. Baier, N. Yokoya, and B. Demir, “Learning convolutional sparse coding on complex domain for interferometric phase restoration,” *IEEE Trans. Neural Netw. Learn. Syst.*, early access, Apr. 9, 2020, doi: 10.1109/TNNLS.2020.2979546.

[46] L. Loncan et al., “Hyperspectral pansharpening: A review,” *IEEE Geosci. Remote Sens. Mag.*, vol. 3, no. 3, pp. 27–46, Sep. 2015.

[47] N. Yokoya, C. Grohnfeldt, and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” *IEEE Geosci. Remote Sens. Mag.*, vol. 5, no. 2, pp. 29–56, Jun. 2017.

[48] N. Yokoya and A. Iwasaki, “Airborne hyperspectral data over Chikusei,” Univ. Tokyo, Tokyo, Japan, Tech. Rep. SAL-2016-05-27, 2016.

[49] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” *ACM Trans. Intell. Syst. Technol.*, vol. 2, no. 3, pp. 1–27, 2011.

<figure><img src="image_11.png" /></figure>

**Naoto Yokoya** (Member, IEEE) received the M.Eng. and Ph.D. degrees from the Department of Aeronautics and Astronautics, The University of Tokyo, Tokyo, Japan, in 2010 and 2013, respectively.

He was an Assistant Professor with the University of Tokyo from 2013 to 2017. From 2015 to 2017, he was an Alexander von Humboldt Fellow with the German Aerospace Center (DLR), Oberpfaffenhofen, Germany, and Technical University of Munich (TUM), Munich, Germany. He is a Lecturer with the University of Tokyo and a Unit Leader with the RIKEN Center for Advanced Intelligence Project, Tokyo, Japan, where he leads the Geoinformatics Unit. His research is focused on the development of image processing, data fusion, and machine learning algorithms for understanding remote sensing images, with applications to disaster management.

Dr. Yokoya received the first place in the 2017 IEEE Geoscience and Remote Sensing Society (GRSS) Data Fusion Contest organized by the Image Analysis and Data Fusion Technical Committee (IADF TC). He was the Co-Chair from 2017 to 2019 of the IEEE GRSS IADF TC and the Chair from 2019 to 2021. He has been the Secretary of the IEEE GRSS All Japan Joint Chapter since 2018. He is an Associate Editor for the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING (JSTARS). He was a Guest Editor for the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING (JSTARS) from 2015 to 2016, the *Remote Sensing* from 2016 to 2020, and the IEEE GEOSCIENCE AND REMOTE SENSING LETTERS (GRSL) from 2018 to 2019.

<figure><img src="image_12.png" /></figure>

**Danfeng Hong** (Member, IEEE) received the M.Sc. degree (summa cum laude) in computer vision from the College of Information Engineering, Qingdao University, Qingdao, China, in 2015, and the Dr.-Ing. degree (summa cum laude) in signal processing in Earth observation (SiPEO), Technical University of Munich (TUM), Munich, Germany, in 2019.

Since 2015, he has been a Research Associate with Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR), Oberpfaffenhofen, Germany. He is a Research Scientist and leads a Spectral Vision Working Group at IMF, DLR, and also an Adjunct Scientist with GIPSA-laboratory, Grenoble INP, CNRS, University of Grenoble Alpes, Grenoble, France. His research interests include signal/image processing and analysis, hyperspectral remote sensing, machine/deep learning, artificial intelligence, and their applications in Earth vision.

<figure><img src="image_13.png" /></figure>

**Jocelyn Chanussot** (Fellow, IEEE) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology (Grenoble INP), Grenoble, France, in 1995, and the Ph.D. degree from the Université de Savoie, Annecy, France, in 1998.

He was a Visiting Scholar with Stanford University, Stanford, CA, USA, KTH, Stockholm, Sweden, and NUS, Singapore. Since 1999, he has been with Grenoble INP. Since 2013, he has been an Adjunct Professor with the University of Iceland, Reykjavik, Iceland. From 2015 to 2017, he was a Visiting Professor with the University of California at Los Angeles (UCLA), Los Angeles, CA, USA. He holds the AXA Chair in remote sensing (RS). He is an Adjunct Professor with the Chinese Academy of Sciences, Aerospace Information Research Institute, Beijing. He is also a Professor of signal and image processing with Grenoble INP. His research interests include image analysis, hyperspectral RS, data fusion, machine learning, and artificial intelligence.

Dr. Chanussot was a member of the Institut Universitaire de France from 2012 to 2017. He was a member of the Machine Learning for Signal Processing Technical Committee of the IEEE Signal Processing Society from 2006 to 2008. He was the Founding President of the IEEE Geoscience and Remote Sensing French Chapter from 2007 to 2010 which received the 2010 IEEE GRS-S Chapter Excellence Award. He has received multiple Outstanding Paper Awards. He was the Vice-President of the IEEE Geoscience and Remote Sensing Society and the In-Charge of meetings and symposia from 2017 to 2019. He was the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution in Remote Sensing (WHISPERS). He was the Chair from 2009 to 2011 and the Co-Chair of the GRS Data Fusion Technical Committee from 2005 to 2008. He was the Program Chair of the IEEE International Workshop on Machine Learning for Signal Processing in 2009. He is an Associate Editor for the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, the IEEE TRANSACTIONS ON IMAGE PROCESSING and the PROCEEDINGS OF THE IEEE. He was the Editor-in-Chief of the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING from 2011 to 2015. In 2014, he served as a Guest Editor for the *IEEE Signal Processing Magazine*. He is a Highly Cited Researcher (Clarivate Analytics/Thomson Reuters, from 2018 to 2019).

<figure><img src="image_14.png" /></figure>

**Jian Kang** (Member, IEEE) received the B.S. and M.E. degrees in electronic engineering from the Harbin Institute of Technology (HIT), Harbin, China, in 2013 and 2015, respectively, and the Dr.-Ing. degree in signal processing in Earth observation (SiPEO) from the Technical University of Munich (TUM), Munich, Germany, in 2019.

In August of 2018, he was a Guest Researcher with the Institute of Computer Graphics and Vision (ICG), TU Graz, Graz, Austria. He is with the Research Institute of Electronic Engineering Technology, HIT, and Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin (TU Berlin), Berlin, Germany. His research focuses on signal processing and machine learning, and their applications in remote sensing (RS). In particular, he is interested in multidimensional data analysis, geophysical parameter estimation based on InSAR data, SAR denoising, and deep learning-based techniques for RS image analysis.

Dr. Kang received first place of the Best Student Paper Award in EUSAR 2018, Aachen, Germany.