# Multimodal GANs: Toward Crossmodal Hyperspectral–Multispectral Image Segmentation

Danfeng Hong, Jing Yao, Deyu Meng, Zongben Xu, Jocelyn Chanussot

<figure><img src="image_1.png" /></figure>

## To cite this version:

Danfeng Hong, Jing Yao, Deyu Meng, Zongben Xu, Jocelyn Chanussot. Multimodal GANs: Toward Crossmodal Hyperspectral–Multispectral Image Segmentation. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (6), pp.5103-5113. 10.1109/TGRS.2020.3020823. hal-03429662

HAL Id: hal-03429662

https://hal.science/hal-03429662v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Multimodal GANs: Toward Crossmodal
Hyperspectral–Multispectral
Image Segmentation

Danfeng Hong<sup>D</sup>, Member, IEEE, Jing Yao<sup>D</sup>, Deyu Meng<sup>D</sup>, Member, IEEE, Zongben Xu, Member, IEEE, and Jocelyn Chanussot<sup>D</sup>, Fellow, IEEE

**Abstract**—This article addresses the problem of semantic segmentation with limited cross-modality data in large-scale urban scenes. Most prior works have attempted to address this issue by using multimodal deep neural networks (DNNs). However, their ability to effectively blending different properties across multimodalities and robustly learning representations from complex scenes remains limited, particularly in the absence of sufficient and well-annotated training images. This leads to a challenge related to cross-modality learning with multimodal DNNs. To this end, we introduce two novel plug-and-play units in the network: self-generative adversarial networks (GANs) module and mutual-GANs module, to learn perturbation-insensitive feature representations and to eliminate the gap between multimodalities, respectively, yielding more effective and robust information transfer. Furthermore, a patch-wise progressive training strategy is devised to enable effective network learning with limited samples. We evaluate the proposed network on two multimodal (hyperspectral and multispectral) overhead image data sets and achieve a significant improvement in comparison with several state-of-the-art methods.

**Index Terms**—Classification, convolutional neural network (CNN), cross modality, deep learning (DL), generative adversarial networks (GANs), hyperspectral (HS), multispectral (MS), remote sensing, semantic segmentation, single image training.

## I. INTRODUCTION

SEMANTIC segmentation is a fundamental but still challenging problem in computer vision, which is of great benefit to many potential applications, such as visual scene

Manuscript received May 24, 2020; revised July 24, 2020; accepted August 28, 2020. This work was supported in part by the National Natural Science Foundation of China under Grant U1811461, Grant 11690011, and Grant 61721002; and in part by the AXA Research Fund. (Corresponding author: Jing Yao.)

Danfeng Hong is with the GIPSA-laboratory, CNRS, Grenoble INP, University of Grenoble Alpes, 38000 Grenoble, France (e-mail: hongdan-feng1989@gmail.com).

Jing Yao and Zongben Xu are with the School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an 710049, China (e-mail: jasonyao@stu.xjtu.edu.cn; zbxu@mail.xjtu.edu.cn).

Deyu Meng is with the School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an 710049, China, and also with the Macau Institute of Systems Engineering, Macau University of Science and Technology, Taipa, Macau (e-mail: dymeng@mail.xjtu.edu.cn).

Jocelyn Chanussot is with the LJK, CNRS, INRIA, Grenoble INP, University of Grenoble Alpes, 38000 Grenoble, France, and also with Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China (e-mail: jocelyn@hi.is).

Color versions of one or more of the figures in this article are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TGRS.2020.3020823

<table><caption>TABLE I
QUALITATIVE COMPARISON OF SEMANTIC SEGMENTATION USING STREET VIEW IMAGES AND OVERHEAD IMAGES</caption><thead><tr><th>Semantic Segmentation</th><th>Street View Images</th><th>Overhead Images</th></tr></thead><tbody><tr><td>Goal</td><td colspan="2">Semantic Segmentation</td></tr><tr><td>Perspective</td><td>Horizontal</td><td>"Bird's"</td></tr><tr><td>Scene Scale</td><td>Small</td><td>Large</td></tr><tr><td>Spatial Resolution</td><td>High</td><td>Low</td></tr><tr><td>Image Quality</td><td>High</td><td>Low</td></tr><tr><td>Data Acquisition</td><td>Easy</td><td>Hard</td></tr><tr><td>Labeled Samples</td><td>Sufficient</td><td>Limited</td></tr><tr><td>Labeling Cost</td><td>Low</td><td>High</td></tr><tr><td>Ground Truth Maps</td><td>Dense</td><td>Sparse</td></tr></tbody></table>

understanding, self-driving, robotic navigation, urban monitoring, and planning, to name a few. It aims to assign a semantic category to each pixel in a studied image scene.

The recent great success of deep convolutional neural networks (CNNs) has made significant progress on semantic segmentation tasks, making it possible to better enhance our understanding and capabilities by semantically segmenting urban scenes with tons of labeled street view images. Yet, it is less investigated on a larger and even global scale due to the inherent limitations of the street view images. On the contrary, the overhead images hold a larger coverage on the surface of the urban area [1], which enables us to detect the large-scale region of interests more easily, due to its bird's view. Although several attempts have been recently made to transfer the existing segmentation networks, e.g., fully convolutional network (FCN) [2], SegNet [3], and U-Net [4], into this task with overhead images, there are many big differences between the two types of images. A qualitative comparison is detailed in Table I to highlight the similarities and differences in semantic segmentation using street view images and overhead images. Please note that overhead images can be regarded as a special case of remote sensing images because of their nearly vertical imaging angle. In our case, to distinguish the differences with street view images of horizontal imaging more clearly and make a more vivid comparison, we adopt the term of "overhead" images more frequently.

It should be noted, however, that unlike street view images with rich contextual information, overhead images inevitably suffer from low spatial resolution and a relatively poor detailed
<figure><img src="image_3.png" /><figcaption>Fig. 1. Illustration to clarify the differences in semantic segmentation of using street view images and overhead images as well as the potential in large-scale urban parsing.</figcaption></figure>

appearance due to the remotely sensed imaging and the limitations of current devices. In addition, labeling overhead images is expensive and they are hardly available on a large scale, compared to street view images. Consequently, classification of overhead images, e.g., hyperspectral (HS) and multispectral (MS), is as often as not performed in the pixelwise or patchwise input fashion [5], [6]. A well-known pixel-based HS image classification technique is morphological profiles (MPs) [7]. MPs and its variants [8]–[10] have shown the effectiveness in extracting spatial–spectral features from HS images for the classification task. Another type of representative pixelwise methods is deep learning (DL)-based classification, e.g., autoencoder [11] and recurrent neural networks (RNNs) [12]. Due to the advancement of CNN-based techniques, the patchwise classification methods have been paid increasing attention to HS or MS image classification in recent years. Some representative and promising CNN-based networks [13]–[19] have been proposed and achieved very competitive classification performance.

## A. Motivation

Fig. 1 shows the to-be-solved challenges and potential solutions for semantic segmentation in large-scale urban scenes by showing the main differences with the images of different views. In detail, we clarify our motivation to answer three “why” questions.

1) *Why parse urban scenes with overhead images?*

2) *Why use multimodal data?*

3) *Why learn compact crossmodal representations?*

1) *From Street to Bird's View:* Different from street view images, overhead images can capture a wider perspective field benefited from a “bird’s view.” This enables a feasible solution for large-scale semantic segmentation.

2) *From Unimodal to Multimodal Data:* Limited by the low spatial resolution and quality of overhead images, unimodal data inevitably meet the bottleneck in a performance gain [20]. Therefore, an alternative to maximize the segmentation accuracy is to jointly leverage the multimodal data to further enhance feature representations.

3) *From Multimodal to Crossmodal Learning:* In reality, the information-rich data are often accompanied by high storage, transmission, and computational cost. In this case,

only the limited multimodal correspondences can be used to train a model, while one modality is absent in the prediction phase. Hence, learning compact and discriminative crossmodal representations is conductive to reasoning and predicting the larger scale semantic segmentation results [21].

## B. Challenges and Objective

1) **Challenges:** Four main challenges for large-scale semantic segmentation with overhead images are summarized as follows.

### Low-Quality

1) **Images:** Remotely sensed imaging (low spatial resolution) and various spectral variabilities caused by complex noise and atmospheric effects.

2) **Few Labeled Data:** Large geospatial coverage and costly labeling.

3) **High Data Heterogeneity:** Different imaging mechanisms, leading to insufficient information fusion.

4) **Weak Modality Transferability:** The absence of certain one modality in model inference.

2) **Objective:** Targeted at these challenges, we aim at improving the representation capability of deep CNNs for large-scale semantic segmentation under the case of using low quality and a few labeled overhead images to yield more compacted and discriminative cross-modal representations.

## C. Contributions

Our contributions are fourfold as follows.

1) Inspired by the recent success of multimodal DL (MDL) and generative adversarial networks (GANs), we propose a GAN-driven cross-modal network, called SM-GANs, that enables the large-scale semantic segmentation on urban scenes at an accuracy level unattainable by existing multimodal FCN frameworks.

2) Two plug-and-play modules, self-GANs and mutual-GANs, are devised in networks to transfer the information across modalities more robustly and effectively.

3) The proposed SM-GANs are equipped with a patch-to-patch progressive learning strategy, making it applicable to network training on a few labeled samples.

4) We validate the superiority and effectiveness of SM-GANs on two multimodal overhead data sets with extensive ablation analysis.

The remainder of this article is organized as follows. Section II presents the related work from three aspects: semantic segmentation in large-scale urban scenes, multimodal/cross-modal deep networks, and adversarial learning via GANs. Section III elaborates the proposed GAN-based network. In Section IV, we show extensive quantitative and qualitative results on two HS-MS data sets. Finally, this article ends with a summary with the future outlook in Section V.

# II. RELATED WORK

## A. Semantic Segmentation in Large-Scale Urban Scenes

Due to the powerful deep CNNs, the research on semantic segmentation has made unprecedented progress
in recent years. Most of these state-of-the-art DL-based frameworks [22]–[25] target at urban scene parsing with street view images. However, a nearly horizontal field of vision makes it difficult to parse a large-scale urban area without extremely diverse training samples (e.g., multiview data). As a feasible and desirable alternative, the overhead images have been garnering growing attention on semantic segmentation [26]–[28], particularly in large-scale urban scenes. Furthermore, considering different characteristics of overhead images (see Table I), we also provide a simple review of current state-of-the-art DL methods for HS or MS image classification. Chen et al. [11] applied stacked autoencoder networks to dimension-reduced HS images for HS image classification. Chen et al. [13] designed CNNs to extract spatial-spectral features more effectively from HS images, thereby yielding higher classification performance. Paoletti et al. [15] proposed a new CNN architecture for fast HS image classification, achieving desirable classification results. The same investigators introduced a promising capsule concept into networks with the application to HS image analysis [14]. Moreover, a deep pyramidal residual network was developed to extract spatial-spectral representations for HS image classification [17]. Haut et al. [16] integrated active learning and CNNs under a new Bayesian framework for HS image classification. Furthermore, RNNs [29] can process the spectral signatures as sequential data. In [12], a cascaded RNN was proposed to make full use of spectral information for high-accuracy HS image classification.

## B. Multimodal/Cross-Modal Deep Networks

Before starting a methodology review, we first clarify the multimodal concept. In general, describing the same object with different information or properties is perceived as multimodal data. They might correspond to different contexts, sensors, resolutions, and so on. More details might be found in [30] as well. MDL proposed by Ngiam et al. [30] is one of the seminal multimodal networks, which aims at finding a joint space where the discriminative feature representation is expected to be learned over multimodalities with multilayered neural networks. Silberer et al. [31] extended the work to a semisupervised version by additionally using a term into loss function that predicts the labels. Similarly, Srivastava and Salakhutdinov [32], [33] considered the multimodal data fusion or learning from the perspective of probabilistic graphical models. Besides, some follow-up works [26], [34]–[38] have been successively proposed by the attempts to increase the capability of information blending in-between multimodalities more effectively and efficiently with different strategies, i.e., multiscale fusion [26], bidirectional symmetrical mechanism [34], convoluted mixture of deep experts (CMoDE) [36], and highly dense connectivity [37]. Nevertheless, their transferability across modalities still remains limited due to insufficient activation on neurons and sensitivity to layer-by-layer accumulation of feature perturbation.

Unlike the modality’s joint representation, another representative method builds the disjunct subnetworks in a coordinated fashion to learn the discriminative features independently

for each modality and couples them by enforcing various structured constraints onto the resulting encoder layers. These structures can be measured by similarity [39], correlation [40], and sequentiality [41].

## C. Adversarial Learning via GANs

There have been many GAN-induced models being explored to enhance the robustness of features by: 1) augmenting the training samples with adversarial examples [42]–[44]; 2) reducing the magnitude of gradients with defensive distillation [45]; and 3) attempting to eliminate the effects of the adversarial noises [46]–[48]. Very recently, Yu et al. [49] proposed to defend against attacks of various adversarial perturbations by generating distinctive “fingerprint” attributes of the input images. Our work is mainly inspired by the recent success of adversarial learning, but beyond them by focusing on modeling the adversarial perturbation into end-to-end multimodal networks for large-scale semantic segmentation.

# III. PROPOSED APPROACH

## A. Overview

The requirements for our problem setting drive us to fit the up-to-date modules into the networks toward robust and discriminative cross-modal representation learning for semantic segmentation in complex large-scale urban scenes. Fig. 2 shows the architecture overview of the proposed network, which follows a dual-SegNet architecture [26] with multimodal reconstruction terms in MDL [30]. More specifically, it starts with a VGG-like configuration [50] as a feature extractor for each modality. The outputs of each modality are fed into the self-GANs module for robust representation against adversarial noise or perturbation. The mutual-GANs module is subsequently activated for highly compact feature blending, before the features of each modality are fed into joint representation. Furthermore, we employ a patch-to-patch progressive learning strategy to make our network training feasible and learnable to the cases of a few training samples.

## B. Self-GANs Module

Due to the internal and external factors (e.g., sensor noises, illumination, and physically and chemically atmospheric effects) [51], [52], it is inevitable to meet the adversarial attacks in imaging products, thereby hurting the network performance. To this end, we employ a self-GANs module to capture the “fingerprint” attributes [49] of sample images robustly against adversarial perturbation. The resulting “fingergprints” (FP) can be defined as

$$ \mathrm{FP}(\mathbf{z}_i) = \mathbf{z}_i - G_i(\mathbf{z}_i) \qquad (1) $$

where $G_i$ denotes the generator for the ith modality and $\mathbf{z}_i$ is the feature map from the VGG-like extractor [50]. Accordingly, $G_i(\mathbf{z}_i)$ represents the self-adversarial output generated by a discriminator $D_i$ to model the adversarial behavior between $G_i(\mathbf{z}_i)$ and $\mathbf{z}_i$. Our robust features, denoted as $g_r(\cdot)$, can be, therefore, obtained as an adaptively weighted product

$$ g_r(\mathbf{z}_i) = [u_r(\mathrm{FP}(\mathbf{z}_i)), v_r(\mathrm{G}(\mathbf{z}_i))] \qquad (2) $$
<figure><img src="image_4.png" /><figcaption>Fig. 2. Overview of our SM-GANs for semantic segmentation using overhead images in large-scale urban scenes.</figcaption></figure>

where $u_r(\text{FP}(\mathbf{z}_i)) = \alpha \text{FP}(\mathbf{z}_i)$ and $v_r(\text{G}(\mathbf{z}_i)) = \beta \text{G}(\mathbf{z}_i)$. $\alpha$ and $\beta$ are to-be-estimated parameters as a tradeoff to weigh different components and can be learned during the network training. In (2), a concatenation operation in the form of $[\cdot, \cdot]$ is applied to integrate structure-enhanced component and perturbation-insensitive “fingerprints” disentangled from original features in a weighted manner. Using this way, output features are not only abundant being able to better predict the category information but also robust against adversarial samples [see Fig. 3(a) for details].

### C. Mutual-GANs Module

It is worth noting that in the layer of multimodal joint representation, massive connections occur in variables from the same modality, but few neurons across the modalities are activated, even if each modality passes through multiple individual hidden layers before being fed into the joint layer. Despite some efforts that made to reduce the gap between the modalities by putting more focus on the fusion of low- and high-level features [34], [36], [37] or the weight sharing across modalities [26], [53], these tentative works often fail to model the global relationship between cross-modal samples.

Inspired by the effectiveness of mutual learning [53], we therefore propose a mutual-GANs module to drag multimodal training pairs into a joint feature space with a consistent data distribution, further yielding better modality alignment and blending. Given the lth layer input features of two modalities, denoted as $\mathbf{z}_{i,l}$ and $\mathbf{z}_{j,l}$, respectively, their representations in the next layer can be written as

$$ \begin{aligned} \mathbf{z}_{i,l+1} &= [f_{i,l}([\mathbf{z}_{i,l}, f_{j,l}(\mathbf{z}_{j,l})]), f_{i,l}([\mathbf{z}_{j,l}, f_{i,l}(\mathbf{z}_{i,l})])] \\ \mathbf{z}_{j,l+1} &= [f_{j,l}([\mathbf{z}_{j,l}, f_{i,l}(\mathbf{z}_{i,l})]), f_{j,l}([\mathbf{z}_{i,l}, f_{j,l}(\mathbf{z}_{j,l})])] \end{aligned} \quad (3) $$

where $f_{\circ,\Delta}$ denotes the forward propagation function (generator) for oth modality in the $\Delta$th layer. Afterward,

<figure><img src="image_5.png" /><figcaption>Fig. 3. Diagrams of two plug-and-play modules in networks. (a) Self-GANs module. (b) Mutual-GANs module.</figcaption></figure>

a discriminator $D_{i,j}$ is employed on multimodal outputs to enforce a consistent distribution. We call such each mutually generative-adversarial procedure as a block of mutual-GANs module, as shown in Fig. 3(b).

### D. Patch-to-Patch Progressive Training

Unlike street view images that are able to be acquired in quantities with densely labeling, overhead images are limited available and labeled sparsely, despite a larger coverage range.
**Algorithm 1 Network Optimization Scheme for SM-GANs**

Non-overlapped

**Require:** sampling for training patches **x**<sub>i</sub> for the *i*th modality, the corresponding one-hot encoded labels **y**, maxIteration = 2, and maxEpoch = 200.

1: Randomly initialize the parameters **W** and **b** of networks using Xavier and Zeros, respectively, the learning rate 5e-4 with 0.98 decay rate, the regularization parameter 0.001, the training batch 10 with 0.9 momentum, and the to-be-estimated adaptive parameters α and β in Eq. (2) with the initialization of 0.5.

2: **for** The iteration number *p* = 0 to maxIteration **do**

3: **for** The epoch number *t* = 0 to maxEpoch **do**

4: Sample a mini-batch for each modality.

5: Obtain the features **z**<sub>i</sub><sup>t</sup> via VGG-like extractor.

6: Calculate “fingerprints” FP(**z**<sub>i</sub><sup>t</sup>) using Eq. (1).

7: Obtain robust features g<sub>r</sub>(**z**<sub>i</sub><sup>t</sup>) by concatenating “fingerprints” as shown in Eq. (2).

8: Calculate mutual features **z**<sub>i,l+1</sub><sup>t</sup> and **z**<sub>j,l+1</sub><sup>t</sup> based on Eq. (3), and obtain the final output via dot product.

9: Update network parameters by jointly optimizing three loss functions in Eq. (4).

10: **end for**

11: **if** *p* = 1 **then**

12: Horizontal sliding sampling with *k* = 10 intervals for training patches and fine tuning networks.

13: **else**

14: Vertical sliding sampling with *k* = 10 intervals for training patches and fine tuning networks.

15: **end if**

16: **end for**

**Ensure:** Network parameters **W** and **b** as well as adaptive parameters α and β.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Examples of patch-based sampling strategy used for our network training.</figcaption></figure>

On the condition of a few samples, we train the network by the means of patchwise input. Moreover, to reduce the redundancy of training samples and enable the network to capture more abundance patterns, we collect training samples in a progressively overlapped sampling way instead of a densely one. Fig. 4 shows the sampling strategy used in our network training. Such a strategy can be intuitively regarded as a distributed fine-tune process, that is, we first train our

<caption>TABLE II</caption>

OAS AND DATA SIZE FOR DIFFERENT TRAINING STRATEGIES IN PAVIA DATA SETS, WHERE *M* IS THE ABBREVIATION OF MBYTE. PLEASE NOTE THAT THE SAME NUMBER OF PATCHES IS USED FOR BOTH RANDOMLY AND OUR PROGRESSIVE SAMPLING

<table><thead><tr><th>Sampling strategy</th><th>Densely</th><th>Randomly</th><th>Progressively (ours)</th></tr></thead><tbody><tr><td>OA</td><td>67.21%</td><td>51.37%</td><td>72.25%</td></tr><tr><td>Data Size</td><td>208160M</td><td>84.7M</td><td>84.7M</td></tr></tbody></table>

network with the nonoverlapped sampling strategy and then finely tuned our network by sliding the sampling windows horizontally or vertically at a *k*-step interval. This training process will stop until the sliding windows touch the edges of the image. In our case, the *k* value is experimentally assigned to 10 to make a tradeoff between information redundancy (high performance) and high-efficient network training.

To show the effectiveness of the proposed way, we make an intuitive comparison for different sampling strategies of collecting training samples, including pixel-by-pixel patch-based densely sampling, randomly one, and our progressively one. Table II lists a quantitative comparison in terms of overall accuracies (OAs) and corresponding data size of network input in Pavia data sets. As can be seen from Table II, although the densely sampling is capable of covering the whole training scene, the overlapped patches inevitably bring extremely redundant, burdening the network training and degrading the network performance. On the contrary, randomly selecting patches would lost the useful information to a great extent, yielding relatively poor performance. As a tradeoff, our progressive strategy achieves a higher OA with less data input by being able to not only consider the global information from the perspective of the whole image but also reduce the information redundancy. Algorithm 1 details the whole optimization scheme of the proposed SM-GANs.

### E. Objective Function

Given the inputs from different modalities **x**<sub>i</sub>, we define the *l*th hidden layer as **z**<sub>i</sub><sup>(l)</sup>. Then, the network parameters can be updated by jointly optimizing the following overall loss function:

$$L = L_l + L_{\text{rec}} + L_{\text{adv}} \quad (4)$$

where $L_l$ is the cross-entropy loss between input patches and patchwise ground-truth maps. In addition, we also consider the reconstruction loss ($L_{\text{rec}}$) for each modality

$$L_{\text{rec}} = \sum_{i=1}^{n} \|\mathbf{x}_i - \hat{\mathbf{x}}_i\|_{F}^{2} \quad (5)$$

where $\hat{\mathbf{x}}_i$ denotes the reconstructed data of $\mathbf{x}_i$. For the adversarial loss ($L_{\text{adv}}$), it consists of two parts. One installs on the self-GANs module and another acts on the mutual-GANs module

$$ \begin{aligned} L_{\text{adv}} &= L_{\text{adv}}^{s} + L_{\text{adv}}^{m} \\ L_{\text{adv}}^{*} &= \max_{D_{i}^{s}} \mathbb{E}[\log(D_{i}^{*}(\mathbf{z}_{i,r}^{*})) + \log(1 - D_{i}^{*}(\mathbf{z}_{i,f}^{*}))] \end{aligned} \quad (6) $$
where * denotes to be either self-GANs (s) or mutual-GANs (m) and $z_{i,r}^*$ and $z_{i,f}^*$ are adversarial pairs of module outputs.

## IV. EXPERIMENTAL RESULTS

### A. Data Description

To verify the superiority of our newly proposed cross-modal network for large-scale semantic segmentation, two overhead multimodal sets are used in experiments.

1) *Pavia Data Sets:* The data sets<fnref n="1" /> consist of bimodal images—MS and HS)—in two different scenes, which are acquired by the ROSIS sensor over the campus of Pavia University and its surrounding city area, Pavia, Italy. The size of two scenes is 610 × 340 pixels and 1096 × 715 pixels, respectively, at a spatial resolution of 1.3 m, where the MS image has four channels, including red (R), green (G), blue (B), and near-infrared (NIR) wavelengths, while the HS image is captured in 102 spectral bands ranging from 430 to 860 nm. The two scenes are annotated with pixel-level labels of nine land cover classes.<fnref n="2" /> To fit our problem setting of cross-modality learning and investigate the transferability of the proposed method, the networks are trained and validated on both MS and HS images in the scene of Pavia University, and only MS image from the scene of Pavia City Center is used to assess the network performance.

2) *Local Climate Zone Data Sets:* The data sets<fnref n="3" /> are composed of six large-scale city scenes: Munich, Berlin, Hong Kong, Paris, Rome, and Sao Paulo, where we select Munich with bimodality (MS and HS images) for training, and the remaining five scenes only with one modality (MS image) are used to test our model. The average size of these images is 7064 × 6528 pixels with a spatial resolution of 10 m and ten channels for the MS image and 244 spectral bands ranging from 400 to 2500 nm for the HS image. We select the class intersection of six cities, eight categories in total.

### B. Experimental Setup

1) *Implementation Details:* Our approach is implemented on the Tensorflow framework [54]. The model is trained on the training set, and the hyperparameters are determined using a grid search on the validation set. In the training phase, we adopt the Adam optimizer with the “poly” learning rate policy [25]. The current learning rate can be updated by multiplying the base one with $(1 - (\text{iter}/\text{maxIter}))^{\text{power}}$, where the base learning rate and power are set to $5e-4$ and 0.98, respectively. We use the convolutional autoencoder to pretrain the subnetworks for each modality to greatly reduce the training time of the model and find a better local optimum easier. Moreover, the momentum is parameterized by 0.9, and the training batch is set to 10. We will stop training when the validation loss fails to decrease. Furthermore, due to the limited training image (only one scene in our case), the patchwise input strategy is adopted to make the network

<table><caption>TABLE III
ABLATION ANALYSIS ON THE PAVIA DATA SETS, WHERE R, S, AND M DENOTE THE RECONSTRUCTION TERM IN (5), SELF-GANs MODULE, AND MUTUAL-GANs MODULE, RESPECTIVELY</caption><thead><tr><th>Model</th><th>R</th><th>S</th><th>M</th><th>OA</th><th>mIoU</th><th>mF<sub>1</sub></th></tr></thead><tbody><tr><td>D-SegNet [26]</td><td>×</td><td>×</td><td>×</td><td>78.25</td><td>57.14</td><td>69.52</td></tr><tr><td>D-SegNet-R</td><td>✓</td><td>×</td><td>×</td><td>81.49</td><td>60.61</td><td>72.60</td></tr><tr><td>S-GANs-R</td><td>✓</td><td>✓</td><td>×</td><td>91.33</td><td>72.37</td><td>82.21</td></tr><tr><td>M-GANs-R</td><td>✓</td><td>×</td><td>✓</td><td>92.74</td><td>75.46</td><td>84.69</td></tr><tr><td>SM-GANs-R</td><td>✓</td><td>✓</td><td>✓</td><td><b>93.30</b></td><td><b>75.92</b></td><td><b>85.07</b></td></tr></tbody></table>

trainable, where the size of patches used for segmentation is uniformly set to 50 × 50 pixels for all compared methods.

2) *Network Configuration:* The proposed SM-GANs consist of four main parts as follows.

1) *VGG-Like Extractor:* We have three convolutional blocks. In each block, there are two $3 \times 3$ convolutional filters, a $2 \times 2$ max-pooling layer, and a ReLU layer. Moreover, 16, 32, and 64 filters are equipped with three convolutional blocks, respectively.

2) *Self-GANs Module:* We have two convolutional blocks. Similarly, there are two $3 \times 3 \times 64$ convolutional filters, a $2 \times 2$ max-pooling layer, and a ReLU layer in each block.

3) *Mutual-GANs Module:* It shares the same network setting with self-GANs module without pooling layers.

4) *Upsampling Module:* It is nothing but a “decoder” architecture of VGG-like extractor with an additional softmax layer connecting the one-hot encoded labels.

3) *Evaluation Metrics:* Three commonly used indices, (pixel-wise) OA, mean Intersection over Union (mIoU), and mean $F_1$ score (m$F_1$), are calculated to quantitatively evaluate the performance of semantic segmentation by collecting all pixelwise predictions.

### C. Ablation Studies

Our network takes the D-SegNet [26] as a backbone. There are three different modules in the proposed network, such as multimodal reconstruction (R), self-GANs module (S), and mutual-GANs module (M). To validate the effectiveness of these components, we perform the ablation analysis on the Pavia data sets.

Table III lists a progressive performance gain by gradually adding different modules. In detail, the use of R module brings around 3% performance improvement in all indices compared to the baseline D-SegNet. More significantly, GAN-based modules outperform the baselines dramatically at an increase of over 10% in terms of OA, mIoU, and $mF_1$. Independently using S or M module with D-SegNet has shown competitive results. However, when jointly embedding S and $M^4$ modules, the proposed SM-GANs are able to capture the discrepancies of learned cross-modal representations more effectively and

<footnote n="4">⁴In our model, we found that two blocks of mutual-GANs used are enough for desirable results, while more blocks would bring additional burdens on computational cost and hurt on network performance.</footnote>

<footnote n="1">¹https://rslab.ut.ac.ir/data</footnote>

<footnote n="2">²To unify the category in the two scenes, we updated some samples from Pavia University using USGS spectral library, making it comparable to those from Pavia City Center.</footnote>

<footnote n="3">³http://www.grss-ieee.org/2017-ieee-grss-data-fusion-contest/</footnote>
<table><caption>TABLE IV
QUANTITATIVE RESULTS ON THE PAVIA DATA SETS. THE BEST ONE IS SHOWN IN BOLD</caption><thead><tr><th>Model</th><th>Asph.</th><th>Mead.</th><th>Tile</th><th>Tree</th><th>Water</th><th>Soil</th><th>Bitu.</th><th>Brick</th><th>Shad.</th><th>OA</th><th>mIoU</th><th>mF<sub>1</sub></th></tr></thead><tbody><tr><td>Baseline</td><td>98.19</td><td>79.13</td><td>60.67</td><td>90.35</td><td>60.43</td><td>66.49</td><td>61.41</td><td>42.83</td><td>99.93</td><td>65.54</td><td>45.44</td><td>59.12</td></tr><tr><td>MDL [30]</td><td>90.77</td><td>88.61</td><td>20.99</td><td>86.25</td><td>98.70</td><td>71.95</td><td>58.83</td><td>41.68</td><td><b>100</b></td><td>70.73</td><td>50.58</td><td>63.18</td></tr><tr><td>Corr-AE [55]</td><td>97.42</td><td>74.89</td><td>66.66</td><td>93.84</td><td>70.44</td><td>68.24</td><td>80.97</td><td>52.92</td><td>99.97</td><td>73.00</td><td>50.04</td><td>63.61</td></tr><tr><td>Deep CCA [56]</td><td>90.33</td><td>82.88</td><td>73.99</td><td>83.23</td><td>83.92</td><td>69.97</td><td>67.13</td><td>32.55</td><td>99.97</td><td>79.33</td><td>53.12</td><td>66.37</td></tr><tr><td>BJRL [34]</td><td><b>98.54</b></td><td>74.69</td><td>74.24</td><td>88.12</td><td>93.13</td><td>61.83</td><td>41.83</td><td>62.01</td><td>99.97</td><td>83.02</td><td>53.49</td><td>67.54</td></tr><tr><td>D-SegNet-R [26]</td><td>95.24</td><td>85.34</td><td>83.68</td><td>89.58</td><td>79.40</td><td>66.83</td><td>70.69</td><td>70.69</td><td>98.22</td><td>81.49</td><td>60.61</td><td>72.60</td></tr><tr><td>LFC [35]</td><td>93.17</td><td>82.75</td><td>62.31</td><td>87.87</td><td>99.87</td><td>63.78</td><td>73.72</td><td>53.18</td><td>99.90</td><td>83.89</td><td>61.40</td><td>73.06</td></tr><tr><td>CMoDE [36]</td><td>96.67</td><td><b>93.59</b></td><td>70.74</td><td>91.56</td><td>99.70</td><td>62.29</td><td>68.81</td><td><b>71.55</b></td><td>99.93</td><td>86.91</td><td>67.00</td><td>78.01</td></tr><tr><td>HyperDense [37]</td><td>97.24</td><td>56.38</td><td>73.17</td><td>97.76</td><td>99.96</td><td>70.73</td><td>79.92</td><td>71.36</td><td>99.93</td><td>88.22</td><td>68.41</td><td>78.78</td></tr><tr><td>SM-GANs-R</td><td>98.22</td><td>66.38</td><td><b>86.87</b></td><td><b>98.21</b></td><td><b>99.97</b></td><td><b>88.17</b></td><td><b>84.82</b></td><td>60.86</td><td>99.93</td><td><b>93.30</b></td><td><b>75.92</b></td><td><b>85.07</b></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Fig. 5. Results of semantic segmentation on the Pavia data sets.</figcaption></figure>

robustly, thereby yielding the best segmentation results with three kinds of different performance measures.

## D. Comparison With State-of-the-Art

Several state-of-the-art baselines related to our cross-modal task are selected for comparison, including baseline.<fnref n="5" />

1) *Multimodal Semantic Labeling*: MDL [30], Corr-AE [55], deep canonical correlation analysis (Deep CCA) [56], and bidirectional joint representation learning (BJRL) [34].

2) *Multimodal Semantic Segmentation*: Dual-stream Seg-Net (D-SegNet) [26], late-fused convolution (LFC) [35], CMoDE [36], and HyperDenseNet [37].

Table IV shows the quantitative performance comparison in terms of pixelwise OA, mIoU, and mean $F_1$ score. Limited by the feature diversity, the baseline yields poor classification performance, while there is a performance improvement (about 5%) in MDL due to the multimodal information transfer. Corr-AE couples two subnetworks by enforcing the structural measurement on hidden layers, i.e., Euclidean similarity, which allows a higher pixelwise classification result by a 2.27% gain in OA, but a nearly similar mIoU and $mF_1$ with MDL. By considering the correlation between samples, Deep CCA is more likely to catch the shared features across modalities and obtains better classification results with a large improvement of 6% over Corr-AE. BJRL behaves superiorly compared to the other models, which is greatly benefited from cross-modal weight sharing between subnetworks.

<footnote n="5">⁵We train a random forest classifier on original pixelwise features.</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 6. Results of semantic segmentation for five cities on the LCZ data sets.</figcaption></figure>

Different from those models for pixelwise semantic labeling, segmentation-based networks pay more attention on modeling short-range spatial relationship between the pixels. Therefore, such a pattern is naturally capable of identifying pixels more accurately than those models that focus on pixelwise labeling. Moreover, as can be seen from Table IV, by making use of convolutional mixture through multimodalities, the performance of CMODE is superior to that of later fusion models (e.g., D-SegNet-R and LFC) at an improvement of 5.42% and 3.02% in OAs, respectively. A more dense cross-modal network—HyDenseNet—further brings increments of 1.31% OA, 1.41% mIoU, and 0.77% mF₁ on the basis of CMODE. As expected, our SM-GANs network remarkably overcomes other competitors either from the whole perspective or per-class performance, especially in mIoU and mF₁ scores (nearly 7% over HyperDenseNet).

## E. Visual Comparison

Fig. 5 shows the segmentation results of ten different networks. Intuitively, some of these methods fail to recognize Water. One possible reason is a complex mixture of feature channels (e.g., Water and Soil) due to the low spatial resolution of overhead images. Nevertheless, our model is capable of making more accurate inferences. This might be well explained by the fact that due to the GAN-induced modules, our SM-GANs tend to seek out important visual and spectral cues from highly mixed materials in the scene for accurate reasoning. Besides, a trend can be figured out, that is, multimodal segmentation networks achieve more smooth parsing results compared to pixelwise semantic labeling. Furthermore, among these segmentation models, the proposed network shows a more realistic semantic parsing, especially in classifying Soil, which is more approaching to the real scene.
<table><caption>TABLE V
QUANTITATIVE RESULTS ON THE LCZ DATA SETS. THE BEST ONE IS SHOWN IN BOLD</caption><thead><tr><th>Model</th><th>compMr</th><th>openMr</th><th>openLr</th><th>laLow</th><th>deTree</th><th>scTree</th><th>loPl</th><th>Water</th><th>OA</th><th>mIoU</th><th>mF<sub>1</sub></th></tr></thead><tbody><tr><td>Baseline</td><td>61.67</td><td>21.08</td><td>11.80</td><td>66.87</td><td>10.87</td><td>–</td><td>15.30</td><td>99.77</td><td>27.47</td><td>24.79</td><td>32.55</td></tr><tr><td>MDL [30]</td><td>57.24</td><td>23.61</td><td>9.63</td><td>70.68</td><td>12.20</td><td>–</td><td>55.74</td><td>99.77</td><td>35.84</td><td>27.55</td><td>36.78</td></tr><tr><td>Corr-AE [55]</td><td>58.54</td><td>30.19</td><td>13.72</td><td>60.28</td><td>16.21</td><td>–</td><td>74.39</td><td><b>99.88</b></td><td>41.34</td><td>30.03</td><td>40.30</td></tr><tr><td>Deep CCA [56]</td><td>59.71</td><td>31.17</td><td>8.90</td><td>63.54</td><td>55.93</td><td>–</td><td>60.60</td><td><b>99.88</b></td><td>47.14</td><td>32.69</td><td>43.44</td></tr><tr><td>BJRL [34]</td><td>60.89</td><td>43.91</td><td>40.55</td><td>50.54</td><td>62.50</td><td><b>2.63</b></td><td>42.92</td><td><b>99.88</b></td><td>51.52</td><td>34.39</td><td>47.81</td></tr><tr><td>D-SegNet-R [26]</td><td>60.23</td><td>49.59</td><td>77.48</td><td>69.47</td><td>73.39</td><td>–</td><td>12.41</td><td>96.07</td><td>56.20</td><td>41.42</td><td>52.48</td></tr><tr><td>LFC [35]</td><td>59.06</td><td><b>54.13</b></td><td>71.57</td><td>70.37</td><td>74.33</td><td>–</td><td>39.99</td><td>96.13</td><td>61.43</td><td>43.94</td><td>56.33</td></tr><tr><td>CMoDE [36]</td><td>60.37</td><td>50.49</td><td>79.08</td><td>68.44</td><td>76.55</td><td>1.56</td><td>43.13</td><td>96.07</td><td>63.56</td><td>46.22</td><td>58.28</td></tr><tr><td>HyperDense [37]</td><td>77.77</td><td>51.06</td><td>73.02</td><td>65.48</td><td>76.77</td><td>1.85</td><td>55.18</td><td>96.07</td><td>66.02</td><td>48.19</td><td>60.48</td></tr><tr><td>SM-GANs-R</td><td><b>85.85</b></td><td>18.18</td><td><b>84.79</b></td><td><b>72.61</b></td><td><b>83.37</b></td><td>2.14</td><td><b>79.20</b></td><td>99.31</td><td><b>72.25</b></td><td><b>51.01</b></td><td><b>61.85</b></td></tr></tbody></table>

## F. Results on the LCZ Data Sets

We evaluate our method in more challenging LCZ data sets. The difficulties mainly lie in two aspects. On the one hand, unlike semantic segmentation on land cover classes, LCZ data seek to parse a more complex urban scene within one pixel at a very low spatial resolution, i.e., 100 m. In this case, robust and discriminative multimodal representations are needed eagerly. We, on the other hand, investigate the networks’ transferability across cities on the LCZ data sets. For this purpose, we select the data sets to perform the experiments as well. As listed in Table V, there is a practically consistent trend in performance gain with that on the Paiva data sets, despite relatively low segmentation accuracies obtained. Similarly, segmentation-based methods observably exceed those compared methods with a focus on pixelwise labeling (averagely 20% increase). Not unexpectedly, the performance of our network is significantly superior to that of others, whose OA is around 6% higher than the method that lies in the second-highest place (see HyperDenseNet).

In addition, qualitative results are also given in Fig. 6 to highlight the visual differences in a large-scale city scene. We select four representative approaches involving our network to display their segmentation results in five different cities. To be more specific, our SM-GANs obtain a more smooth and detailed appearance in urban scene parsing in comparison with other methods, due to its use of adversarial mechanism to minimize the effects of perturbation and mutual learning to eliminate the gap between modalities.

## V. CONCLUSION

In this article, we introduce two plug-and-play GANs-based modules, namely, self-GANs and mutual-GANs, into networks to enable the semantic segmentation in large-scale urban scenes with multimodal overhead images. To address the issue of small-sample learning in network training, an effective patch-to-patch progressive updating strategy is employed. The comprehensive ablation studies have demonstrated that both GAN-guided modules have learned robust features against noise perturbation and more compact representations being able to cross modalities. However, the design to enhance the transferability between modalities in networks is noteworthy

and can be regarded as our future works. In addition, we have to admit that the research on multimodal remote sensing data has been garnering growing attention in the remote sensing community. The high-quality benchmark data sets are the main bottlenecks holding back its development. To this end, we would also like to build more data sets related to this multimodality/cross-modality learning and classification tasks in our future work.

## ACKNOWLEDGMENT

The authors would like to thank Prof. Paolo Gamba (http://tlclab.unipv.it/?/) from the University of Pavia for providing the hyperspectral data of Pavia University and City Center in our experiments and the IEEE GRSS IADF TC for sharing the LCZ data packages used in this article.

## REFERENCES

[1] J. Kang, D. Hong, J. Liu, G. Baier, N. Yokoya, and B. Demir, "Learning convolutional sparse coding on complex domain for interferometric phase restoration," *IEEE Trans. Neural Netw. Learn. Syst.*, early access, Apr. 9, 2020, doi: 10.1109/TNNLS.2020.2979546.

[2] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jun. 2015, pp. 3431–3440.

[3] V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A deep convolutional encoder-decoder architecture for image segmentation," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 39, no. 12, pp. 2481–2495, Dec. 2017.

[4] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in *Proc. MICCAI*. Cham, Switzerland: Springer, 2015, pp. 234–241.

[5] M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza, "Deep learning classifiers for hyperspectral imaging: A review," *ISPRS J. Photogramm. Remote Sens.*, vol. 158, pp. 279–317, Dec. 2019.

[6] B. Rasti et al., "Feature extraction for hyperspectral imagery: The evolution from shallow to deep (overview and toolbox)," *IEEE Geosci. Remote Sens. Mag.*, early access, Apr. 29, 2020, doi: 10.1109/MGRS.2020.2979764.

[7] J. A. Benediktsson, J. A. Palmason, and J. R. Sveinsson, "Classification of hyperspectral data from urban areas based on extended morphological profiles," *IEEE Trans. Geosci. Remote Sens.*, vol. 43, no. 3, pp. 480–491, Mar. 2005.

[8] P. Ghamisi, M. Dalla Mura, and J. A. Benediktsson, "A survey on spectral-spatial classification techniques based on attribute profiles," *IEEE Trans. Geosci. Remote Sens.*, vol. 53, no. 5, pp. 2335–2353, May 2015.

[9] P. Ghamisi, J. A. Benediktsson, G. Cavallaro, and A. Plaza, "Automatic framework for spectral-spatial classification based on supervised feature extraction and morphological attribute profiles," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 7, no. 6, pp. 2147–2160, Jun. 2014.
[10] D. Hong, X. Wu, P. Ghamisi, J. Chanussot, N. Yokoya, and X. X. Zhu, "Invariant attribute profiles: A spatial-frequency joint feature extractor for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 6, pp. 3791-3808, Jun. 2020.

[11] Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, "Deep learning-based classification of hyperspectral data," *IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.*, vol. 7, no. 6, pp. 2094-2107, Jun. 2014.

[12] R. Hang, Q. Liu, D. Hong, and P. Ghamisi, "Cascaded recurrent neural networks for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 8, pp. 5384-5394, Aug. 2019.

[13] Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, "Deep feature extraction and classification of hyperspectral images based on convolutional neural networks," *IEEE Trans. Geosci. Remote Sens.*, vol. 54, no. 10, pp. 6232-6251, Oct. 2016.

[14] M. E. Paoletti et al., "Capsule networks for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 4, pp. 2145-2160, Apr. 2019.

[15] M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza, "A new deep convolutional neural network for fast hyperspectral image classification," *ISPRS J. Photogramm. Remote Sens.*, vol. 145, pp. 120-147, Nov. 2018.

[16] J. M. Haut, M. E. Paoletti, J. Plaza, J. Li, and A. Plaza, "Active learning with convolutional neural networks for hyperspectral image classification using a new Bayesian approach," *IEEE Trans. Geosci. Remote Sens.*, vol. 56, no. 11, pp. 6440-6461, Nov. 2018.

[17] M. E. Paoletti, J. M. Haut, R. Fernandez-Beltran, J. Plaza, A. J. Plaza, and F. Pla, "Deep pyramidal residual networks for spectral-spatial hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 2, pp. 740-754, Feb. 2019.

[18] R. Hang, Z. Li, P. Ghamisi, D. Hong, G. Xia, and Q. Liu, "Classification of hyperspectral and LiDAR data using coupled CNNs," *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 7, pp. 4939-4950, Jul. 2020.

[19] X. Wu, W. Li, D. Hong, J. Tian, R. Tao, and Q. Du, "Vehicle detection of multi-source remote sensing data using active fine-tuning network," *ISPRS J. Photogramm. Remote Sens.*, vol. 167, pp. 39-53, Sep. 2020.

[20] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, "CoSpace: Common subspace learning from hyperspectral-multispectral correspondences," *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 7, pp. 4349-4359, Jul. 2019.

[21] D. Hong, N. Yokoya, N. Ge, J. Chanussot, and X. X. Zhu, "Learnable manifold alignment (LeMA): A semi-supervised cross-modality learning framework for land cover and land use classification," *ISPRS J. Photogramm. Remote Sens.*, vol. 147, pp. 193-205, Jan. 2019.

[22] F. Yu and V. Koltun, "Multi-scale context aggregation by dilated convolutions," 2015, arXiv:1511.07122. [Online]. Available: http://arxiv.org/abs/1511.07122

[23] F. Xia, P. Wang, L. Chen, and A. L. Yuille, "Zoom better to see clearer: Human and object parsing with hierarchical auto-zoom net," in *Proc. ECCV*. Cham, Switzerland: Springer, 2016, pp. 648-663.

[24] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, "Pyramid scene parsing network," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jul. 2017, pp. 2881-2890.

[25] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 40, no. 4, pp. 834-848, Apr. 2018.

[26] N. Audebert, B. L. Saux, and S. Lefèvre, "Semantic segmentation of earth observation data using multimodal and multi-scale deep networks," in *Proc. ACCV*. Cham, Switzerland: Springer, 2016, pp. 180-196.

[27] G. Mattyus, S. Wang, S. Fidler, and R. Urtasun, "HD maps: Fine-grained road segmentation by parsing ground and aerial images," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)*, Jun. 2016, pp. 3611-3619.

[28] L. Zhang et al., "Learning deep structured active contours end-to-end," in *Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.*, Jun. 2018, pp. 8877-8885.

[29] H. Wu and S. Prasad, "Convolutional recurrent neural networks for hyperspectral data classification," *Remote Sens.*, vol. 9, no. 3, p. 298, Mar. 2017.

[30] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep learning," in *Proc. ICML*, 2011, pp. 689-696.

[31] C. Silberer, V. Ferrari, and M. Lapata, "Visually grounded meaning representations," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 39, no. 11, pp. 2284-2297, Nov. 2017.

[32] N. Srivastava and R. Salakhutdinov, "Learning representations for multimodal data with deep belief nets," in *Proc. ICML Workshop*, vol. 79, 2012.

[33] N. Srivastava and R. Salakhutdinov, "Multimodal learning with deep Boltzmann machines," in *Proc. NIPS*, 2012, pp. 2222-2230.

[34] V. Vukotić, C. Raymond, and G. Gravier, "Bidirectional joint representation learning with symmetrical deep neural networks for multimodal and crossmodal applications," in *Proc. ACM Int. Conf. Multimedia Retr. (ICMR)*, 2016, pp. 343-346.

[35] A. Valada, G. L. Oliveira, T. Brox, and W. Burgard, "Deep multispectral semantic scene understanding of forested environments using multimodal fusion," in *Proc. Int. Symp. Exp. Robot. (ISER)*. Cham, Switzerland: Springer, 2016, pp. 465-477.

[36] A. Valada, J. Vertens, A. Dhall, and W. Burgard, "AdapNet: Adaptive semantic segmentation in adverse environmental conditions," in *Proc. IEEE Int. Conf. Robot. Autom. (ICRA)*, May 2017, pp. 4644-4651.

[37] J. Dolz, K. Gopinath, J. Yuan, H. Lombaert, C. Desrosiers, and I. Ben Ayed, "HyperDense-net: A hyper-densely connected CNN for multimodal image segmentation," *IEEE Trans. Med. Imag.*, vol. 38, no. 5, pp. 1116-1126, May 2019.

[38] D. Hong et al., "More diverse means better: Multimodal deep learning meets remote-sensing imagery classification," *IEEE Trans. Geosci. Remote Sens.*, early access, Aug. 24, 2020, doi: 10.1109/TGRS.2020.3016820.

[39] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, "Graph convolutional networks for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, early access, Aug. 18, 2020, doi: 10.1109/TGRS.2020.3015157.

[40] S. Chandar, M. M. Khapra, H. Larochelle, and B. Ravindran, "Correlative neural networks," *Neural Comput.*, vol. 28, no. 2, pp. 257-285, Feb. 2016.

[41] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, "Order-embeddings of images and language," 2015, arXiv:1511.06361. [Online]. Available: http://arxiv.org/abs/1511.06361

[42] C. Szegedy et al., "Intriguing properties of neural networks," 2013, arXiv:1312.6199. [Online]. Available: http://arxiv.org/abs/1312.6199

[43] I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," 2014, arXiv:1412.6572. [Online]. Available: http://arxiv.org/abs/1412.6572

[44] M. Mathieu, J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun, "Disentangling factors of variation in deep representation using adversarial training," in *Proc. NIPS*, 2016, pp. 5040-5048.

[45] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, "Distillation as a defense to adversarial perturbations against deep neural networks," in *Proc. IEEE Symp. Secur. Privacy (SP)*, May 2016, pp. 582-597.

[46] I. Goodfellow et al., "Generative adversarial nets," in *Proc. NIPS*, 2014, pp. 2672-2680.

[47] P. Samangouei, M. Kabkab, and R. Chellappa, "Defense-GAN: Protecting classifiers against adversarial attacks using generative models," 2018, arXiv:1805.06605. [Online]. Available: http://arxiv.org/abs/1805.06605

[48] D. Hong, N. Yokoya, G.-S. Xia, J. Chanussot, and X. X. Zhu, "X-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing data," *ISPRS J. Photogramm. Remote Sens.*, vol. 167, pp. 12-23, Sep. 2020.

[49] N. Yu, L. Davis, and M. Fritz, "Attributing fake images to GANs: Learning and analyzing GAN fingerprints," in *Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV)*, Oct. 2019, pp. 7556-7566.

[50] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," 2014, arXiv:1409.1556. [Online]. Available: http://arxiv.org/abs/1409.1556

[51] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, "An augmented linear mixing model to address spectral variability for hyperspectral unmixing," *IEEE Trans. Image Process.*, vol. 28, no. 4, pp. 1923-1938, Apr. 2019.

[52] D. Hong, N. Yokoya, J. Chanussot, J. Xu, and X. X. Zhu, "Learning to propagate labels on graphs: An iterative multitask regression framework for semi-supervised hyperspectral dimensionality reduction," *ISPRS J. Photogramm. Remote Sens.*, vol. 158, pp. 35-49, Dec. 2019.

[53] X. Nie, J. Feng, and S. Yan, "Mutual learning to adapt for joint human parsing and pose estimation," in *Proc. ECCV*, 2018, pp. 502-517.

[54] M. Abadi et al., "TensorFlow: A system for large-scale machine learning," in *Proc. OSDI*, vol. 16, 2016, pp. 265-283.

[55] F. Feng, X. Wang, and R. Li, "Cross-modal retrieval with correspondence autoencoder," in *Proc. ACM Int. Conf. Multimedia (MM)*, 2014, pp. 7-16.

[56] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, "Deep canonical correlation analysis," in *Proc. ICML*, 2013, pp. 1247-1255.
<figure><img src="image_9.png" /></figure>

**Danfeng Hong** (Member, IEEE) received the M.Sc. degree (summa cum laude) in computer vision from the College of Information Engineering, Qingdao University, Qingdao, China, in 2015, and the Dr. Ing degree (summa cum laude) from Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM), Munich, Germany, in 2019.

Since 2015, he has been a Research Associate with Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR), Oberpfaffenhofen, Germany. He is a Research Scientist and leads the Spectral Vision Working Group at IMF, DLR, and an Adjunct Scientist with the GIPSA-laboratory, CNRS, Grenoble INP, University of Grenoble Alpes, Grenoble, France. His research interests include signal/image processing and analysis, hyperspectral remote sensing, machine/deep learning, artificial intelligence, and their applications in Earth vision.

<figure><img src="image_10.png" /></figure>

**Jing Yao** received the B.Sc. degree from Northwest University, Xi’an, China, in 2014. He is pursuing the Ph.D. degree with the School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an.

From 2019 to 2020, he was a Visiting Student with Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM), Munich, Germany, and the Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR), Oberpfaffenhofen, Germany. His research interests include low-rank modeling, hyperspectral

image analysis, and deep learning-based image processing methods.

<figure><img src="image_11.png" /></figure>

**Deyu Meng** (Member, IEEE) received the B.Sc., M.Sc., and Ph.D. degrees from Xi’an Jiaotong University, Xi’an, China, in 2001, 2004, and 2008, respectively.

From 2012 to 2014, he took his two-year sabbatical leave at Carnegie Mellon University, Pittsburgh, PA, USA. He is a Professor with the School of Mathematics and Statistics, Xi’an Jiaotong University, and an Adjunct Professor with the Faculty of Information Technology, Macau University of Science and Technology, Taipa, Macau. His research

interests include model-based deep learning, variational networks, and meta-learning.

<figure><img src="image_12.png" /></figure>

**Zongben Xu** (Member, IEEE) received the Ph.D. degree in mathematics from Xi’an Jiaotong University, Xi’an, China, in 1987.

He is the Director of the Institute for Information and System Sciences, Xi’an Jiaotong University. His research interests include intelligent information processing and applied mathematics.

Dr. Xu was elected as a member of the Chinese Academy of Science in 2011. He was a recipient of the National Natural Science Award of China in 2007 and the Winner of the CSIAM Su Buchin

Applied Mathematics Prize in 2008. He delivered a speech at the International Congress of Mathematicians 2010. He serves as the Chief Scientist for the National Basic Research Program of China (973 Project).

<figure><img src="image_13.png" /></figure>

**Jocelyn Chanussot** (Fellow, IEEE) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology (Grenoble INP), Grenoble, France, in 1995, and the Ph.D. degree from the Université de Savoie, Annecy, France, in 1998.

Since 1999, he has been with Grenoble INP, where he is a Professor of signal and image processing. He has been a Visiting Scholar with Stanford University, Stanford, CA, USA; the KTH Royal Institute of Technology, Stockholm, Sweden; and the National University of Singapore, Singapore. Since 2013, he has been an Adjunct Professor with the University of Iceland, Reykjavik, Iceland. From 2015 to 2017, he was a Visiting Professor with the University of California at Los Angeles (UCLA), Los Angeles, CA, USA. He holds the AXA Chair in remote sensing and is an Adjunct Professor with Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China. His research interests include image analysis, hyperspectral remote sensing, data fusion, machine learning, and artificial intelligence.

Dr. Chanussot was a member of the Institut Universitaire de France from 2012 to 2017. He was a member of the Machine Learning for Signal Processing Technical Committee of the IEEE Signal Processing Society from 2006 to 2008 and the Program Chair of the IEEE International Workshop on Machine Learning for Signal Processing in 2009. He has received multiple outstanding paper awards. He was the founding President of the IEEE Geoscience and Remote Sensing French Chapter from 2007 to 2010, which received the 2010 IEEE GRS-S Chapter Excellence Award. He was the Vice-President of the IEEE Geoscience and Remote Sensing Society, in charge of meetings and symposia from 2017 to 2019. He was the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution in Remote sensing (WHISPERS). He was the Chair and the Co-Chair of the GRS Data Fusion Technical Committee from 2005 to 2008 and from 2009 to 2011, respectively. He is also an Associate Editor of the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, the IEEE TRANSACTIONS ON IMAGE PROCESSING, and the PROCEEDINGS OF THE IEEE. He was the Editor-in-Chief of the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING from 2011 to 2015. In 2014, he served as a Guest Editor for the *IEEE Signal Processing Magazine*. He was a Highly Cited Researcher (Clarivate Analytics/Thomson Reuters) from 2018 to 2019.