# Learning-Shared Cross-Modality Representation Using Multispectral-LiDAR and Hyperspectral Data

Danfeng Hong, Jocelyn Chanussot, Naoto Yokoya, Jian Kang, Xiao Xiang Zhu

► To cite this version:

Danfeng Hong, Jocelyn Chanussot, Naoto Yokoya, Jian Kang, Xiao Xiang Zhu. Learning-Shared Cross-Modality Representation Using Multispectral-LiDAR and Hyperspectral Data. IEEE Geoscience and Remote Sensing Letters, 2020, 17 (8), pp.1470-1474. 10.1109/LGRS.2019.2944599. hal-03142169

HAL Id: hal-03142169

https://hal.science/hal-03142169v1

Submitted on 28 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Learning-Shared Cross-Modality Representation Using Multispectral-LIDAR and Hyperspectral Data

Danfeng Hong<sup>D</sup>, Student Member, IEEE, Jocelyn Chanussot<sup>D</sup>, Fellow, IEEE, Naoto Yokoya<sup>D</sup>, Member, IEEE, Jian Kang<sup>D</sup>, Student Member, IEEE, and Xiao Xiang Zhu<sup>D</sup>, Senior Member, IEEE

**Abstract**—Due to the ever-growing diversity of the data source, multimodality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multimodalities that exist in both training and test sets, yet they are less investigated in the absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multimodalities in the training process. By this way, the out-of-sample from any of multimodalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multimodal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Light Detection and Ranging (LIDAR) and hyperspectral data set provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.

**Index Terms**—Cross-modality, feature learning, hyperspectral, multimodality, multispectral-Light Detection and Ranging (LIDAR), shared subspace learning.

## I. INTRODUCTION

REMOTE sensing (RS) is one of the most common ways to extract relevant information about Earth and

Manuscript received January 31, 2019; revised June 13, 2019; accepted September 5, 2019. Date of publication January 30, 2020; date of current version July 22, 2020. This work was supported in part by the German Research Foundation (DFG) under Grant ZH 498/7-2, in part by the Helmholtz Association under the framework of the Young Investigators Group SiPEO (VH-NG-1018), and in part by the European Research Council (ERC) under the European Unions Horizon 2020 Research and Innovation Program (Grant agreement No. ERC-2016-StG-714087, Acronym: So2Sat). The work of Naoto Yokoya was supported by the Japan Society for the Promotion of Science (KAKENHI) under Grant 18K18067. (Corresponding author: Xiao Xiang Zhu.)

Danfeng Hong and Xiao Xiang Zhu are with the Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR), 82234 Wessling, Germany, and also with Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM), 80333 Munich, Germany (e-mail: danfeng.hong@dlr.de; xiaoxiang.zhu@dlr.de).

Jocelyn Chanussot is with the Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France, and also with the Faculty of Electrical and Computer Engineering, University of Iceland, Reykjavik 101, Iceland (e-mail: jocelyn@hi.is).

Naoto Yokoya is with the Geoinformatics Unit, RIKEN Center for Advanced Intelligence Project (AIP), RIKEN, Tokyo 103-0027, Japan (e-mail: naoto.yokoya@riken.jp).

Jian Kang is with the Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM), 80333 Munich, Germany (e-mail: jian.kang@tum.de).

Color versions of one or more of the figures in this letter are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/LGRS.2019.2944599

our environment. RS acquisitions can be done by both active [synthetic aperture radar, Light Detection and Ranging (LIDAR)] and passive (optical and thermal range, multispectral, and hyperspectral) devices. The complementary of the data acquired by different platforms can be helpful for more accurately characterizing land use [1]. In this letter, we will focus on the joint use of multispectral-Lidar (MS-Lidar) data, providing detailed information about the ground elevation, and hyperspectral image (HSI), providing information of the physical nature of the sensed materials, using a general cross-modality learning (CML) framework [2].

Intuitively, most existing multimodality feature learning methods basically follow the concatenation-based fusion strategy [3]. However, either early fusion or latter one might be incapable of effectively addressing the aforementioned challenge, as there is a lack of completely paired multimodality samples in the whole data set. The problem setting naturally motivates us to find a latent shared feature space by learning modality-specific projections from the training samples.

For this purpose, some tentative works have been proposed by joint dimensionality reduction or alignment learning. In [2], principal component analysis (PCA) is used to simultaneously project multimodal data into a common subspace. Rasti *et al.* [4] fused the HSI and Lidar data using total variation component analysis for land-cover and land-use mapping. Hong *et al.* [5] jointly embedded the spatial-spectral information for HSI classification. Besides, manifold alignment (MA) has been proven to be another powerful solution. Following it, Tuia *et al.* [6] proposed to align multiview RS imagery on manifolds to reduce the gap between multimodalities. More generally, Banerjee and Buddhiraju [7] transferred the samples of source and target domains into a shared latent domain where the learned features in both domains are expected to be consistent. Du *et al.* [8] for the first time took multimodal RS data analysis as an unsupervised multitask learning problem and proposed a state-of-the-art and milestone blind source separation algorithm for multimodal and HS data processing. Although these methods mentioned above might provide a feasible way for the CML-related issues, the ability to extract the discriminative features remains limited. This possibly results from the lack of directly modeling the latent subspace and label information.

To facilitate the improvement of feature discrimination, we propose to simultaneously learn the shared subspace
<figure><img src="image_2.png" /><figcaption>Fig. 1. Illustration of the proposed cross-modality feature learning. The CML problem specifically refers to model learning using multimodalities in the training phase and testing the model only using one of the multimodalities (please see Section IIA for more details).</figcaption></figure>

and regress the labels from the learned subspace in a joint fashion. Inspired by MA, we also enforce a graph-based alignment constraint on the multimodal data, aiming at more compact subspace learning. In fact, the proposed method in this letter is an extended version of common subspace learning (CoSpace) presented in [2]. The main differences lie in two aspects. On the one hand, we emphatically analyze the effects of different regression strategies, such as ridge regression (ℓ₂-penalty) and sparse regression (ℓ₁-penalty). On the other hand, we further investigate the potentials of CoSpace-based models while handling the heterogeneous data (e.g., MS-Lidar and HS).

## II. METHODOLOGY

Fig. 1 illustrates the proposed cross-modality feature learning framework. In the section, we start with a review of the existing CoSpace model and then discuss and analyze the potentials of using sparse regression in CoSpace. Finally, an optimizer based on the alternating direction method of multipliers (ADMM) is briefly introduced to solve the extended CoSpace.

### A. Brief Motivation

Although operational optical satellites, e.g., Sentinel-2 and Landsat-8, enable the MS data openly and largely available on a global scale, the MS data fail to distinguish similar classes due to its few spectral bands. Rather, the HSI is acquired with rich spectral information, enabling identification of the materials more easily and accurately, but its space coverage is far narrower than that of MS data. This naturally motivates us to investigate a general but interesting question—can a limited amount of HS data partially overlapping with MS data improve the classification performance of the extra large-scale and nonoverlapped MS data? This is a typical CML-based problem setting.

### B. Review of CoSpace

For this purpose, we proposed a feasible solution in [2], namely, ℓ₂-CoSpace. The proposed method, supporting a multimodal input in the training phase, aims at learning a common subspace from multimodalities, where the learned features are expected to be discriminative by fusing the different modality-specific information as much as possible. Theoretically speaking, through the shared feature space, the different modalities can be arbitrarily translated to each other. We also connected the learned features and the label information by means of regression techniques for a more discriminative representation. Moreover, simultaneously considering the above strategies leads to the following joint model.

Given two modalities $X_1 \in \mathbb{R}^{d_1 \times N}$ and $X_2 \in \mathbb{R}^{d_2 \times N}$, namely, HS data with $d_1$ bands by $N$ pixels and MS-LIDAR data with $d_2$ bands by $N$ pixels in our case, the CoSpace can be modeled by optimizing the following objective function:

$$ \min_{\mathbf{P}, \Theta} \left\{ \frac{1}{2} \|\tilde{\mathbf{Y}} - \mathbf{P}\Theta\tilde{\mathbf{X}}\|_{F}^{2} + \frac{\alpha}{2} \|\mathbf{P}\|_{F}^{2} + \frac{\beta}{2} \text{tr}(\Theta\tilde{\mathbf{X}}\mathbf{L}(\Theta\tilde{\mathbf{X}})^{\mathrm{T}}) \right\} \quad (1) $$

where $\mathbf{Y} \in \mathbb{R}^{L \times N}$ denotes the one-hot encoded label matrix and $\tilde{\mathbf{Y}}$ is defined as $[\mathbf{Y}, \mathbf{Y}] \in \mathbb{R}^{L \times 2N}$. $\tilde{\mathbf{X}} = \begin{bmatrix} \mathbf{X}_1 & \mathbf{0} \\ \mathbf{0} & \mathbf{X}_2 \end{bmatrix} \in \mathbb{R}^{(d_1+d_2) \times 2N}$ and $\Theta = [\Theta_1, \Theta_2] \in \mathbb{R}^{d \times (d_1+d_2)}$ represents the subspace projection with respect to $\mathbf{X}_1$ and $\mathbf{X}_2$. $d$ is the dimension of the learned subspace. The variable $\mathbf{P} \in \mathbb{R}^{L \times d}$ is the regression matrix regularized by ℓ₂-norm, which connects the latent subspace and label information for a discriminative feature representation. Moreover, $\mathbf{L} = \mathbf{D} - \mathbf{W} \in \mathbb{R}^{2N \times 2N}$ is defined as a joint Laplacian matrix, and $\mathbf{W}$ is the
<table><caption>**Algorithm 1 CoSpace-based solution**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>Ỹ, X̃, L, and parameters α, β, maxIter.</td></tr><tr><td>Output:</td><td>P, Θ</td></tr><tr><td>1</td><td>t = 1, ζ = 1e-4;</td></tr><tr><td>2</td><td>Initializing P and Θ</td></tr><tr><td>3</td><td>while not converged or t > maxIter do</td></tr><tr><td>4</td><td>Fix other variables to update P</td></tr><tr><td>5</td><td>Fix other variables to update Θ</td></tr><tr><td>6</td><td>Compute the objective function value E<sup>t+1</sup> and check</td></tr><tr><td>7</td><td>the convergence condition: if |E<sup>t+1</sup> - E<sup>t</sup>| / E<sup>t</sup> &lt; ζ then</td></tr><tr><td>8</td><td>Stop iteration;</td></tr><tr><td>9</td><td>else</td></tr><tr><td>10</td><td>|t - t + 1|;</td></tr><tr><td>11</td><td>end</td></tr></tbody></table>

corresponding adjacency matrix formulated as follows:

$$W^{i,j} = \begin{cases} 1/N_k, & \text{if } X^i \text{ and } X^j \text{ are the } k\text{th class} \\ 0, & \text{otherwise} \end{cases} \quad (2)$$

and then **D** is computed by $D^{ii} = \sum_{i \neq j} W^{i,j}$.

For the model solution in (1), we adopt an iterative alternating optimization strategy [9] to convert the nonconvexity of (1) to the convex subproblems of each variable **P** and **Θ**. The optimization subproblem with respect to the variable **P** is a typical least-squares problem with Tikhonov regularization, which has an analytical solution of $P = (\tilde{Y}Q^T)(QQ^T + \alpha I)^{-1}$, where $Q = \Theta X̃$. For the optimization problem of **Θ**, it can be effectively and efficiently solved by ADMM. Please refer to [2] for more details.

### C. Sparse Regression-Based CoSpace ($\ell_1$-CoSpace)

To further improve the CoSpace's representation ability, we propose to model the sparsity-promoting regression matrix, yielding a $\ell_1$-CoSpace. Unlike the original CoSpace model with $\ell_2$-regularized ridge regression ($\ell_2$-CoSpace for short), $\ell_1$-CoSpace learns a sparse regression matrix to connect the latent subspace and label space. More specifically, the advantages of $\ell_1$-norm over $\ell_2$-norm can be summarized as follows.

1) Compared to the $\ell_2$-norm, it is well known that the $\ell_1$-norm plays a role of feature selection, which makes the learned features more robust and further enhances the model's generalization ability.

2) As introduced in [10], the sparsity-based learning or regression technique is capable of better interpreting the intrinsic structure of the data (or feature) space. This might effectively excavate or discover the underlying correspondences between selected features and certain classes, thereby yielding more effective feature learning.

Accordingly, the resulting $\ell_1$-CoSpace can be formulated as

$$\min_{P, \Theta} \left\{ \frac{1}{2} \|\tilde{Y} - P\Theta\tilde{X}\|_{F}^{2} + \alpha \|P\|_{1,1} + \frac{\beta}{2} \text{tr}(\Theta\tilde{X}L(\Theta\tilde{X})^{T}) \right\} \quad (3)$$

<figure><img src="image_3.png" /><figcaption>Fig. 2. False-color images of the used two data. (a) HS data. (b) MS-Lidar data.</figcaption></figure>

where $\|P\|_{1,1} \equiv \sum_{k=1}^{N} \|p_k\|_1$ is used to approximate the sparsity.

Similar to $\ell_2$-CoSpace, the problem (4) can be separated into two convex subproblems for the variables **P** and **Θ**, respectively. Moreover, the optimization problem of **P** can be quickly solved by the well-known soft threshold operator [11] under the ADMM framework, while the solution for the variable **Θ** is same with that in $\ell_2$-CoSpace. Algorithm 1 details the specific solutions for the problem (1) or (4).

## III. EXPERIMENTS

To assess the performance of CoSpace-related methods (e.g., $\ell_2$-CoSpace, $\ell_1$-CoSpace) compared to several state-of-the-art baselines, we explore the classification as a potential application in terms of Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient ($\kappa$). Two popular classifiers: linear support vector machines (LSVM) and canonical correlation forests (CCF) [12], are used in our case.

### A. Data Description

We conducted the experiments on MS-LIDAR and HS data provided by the 2018 IEEE GRSS data fusion contest (DFC2018) [13], where the MS-LIDAR data and HSI were acquired by an Optech Titam MW (14SEN/CON340) with a LiDAR sensor and an ITRES CASI 1500 sensor, respectively. The MS-LIDAR data were collected from three different wavelengths (1550, 1064, and 532 nm) at a 50-cm ground sampling distance (GSD). It consists of 1202 × 4768 pixels with nine bands (three downsampled RGB bands, three intensity bands, and three digital elevation model (DEM) bands). Note that these bands are stacked [14] as the model input. The corresponding HSI with the dimensions of 601 × 2384 × 48 covers the wavelength range from 380 to 1050 nm at a GSD of 1 m. The false-color images for the two used data are shown in Fig. 2.

### B. Experimental Configuration

All models involved in performance comparison are trained on MS-Lidar and HS data and tested only in the presence of MS-LIDAR data to meet our CML's problem setting. To have pixel-to-pixel aligned on the two different modalities, we downsampled the MS-LIDAR and ground truth to the HSI's spatial resolution using the nearest neighbor interpolation. Notably, in the used data set, the sample distribution between the classes is extremely unbalanced. To provide a more reasonable and meaningful performance analysis, we evenly select
<figure><img src="image_4.png" /><figcaption>Fig. 3. Classification maps predicted by different methods under the two classifiers (LSVM and CCF).</figcaption></figure>

200 samples from each class for training<fnref n="1" /> and the rest samples for testing. Furthermore, ten replications were performed for the selection of training and test samples and their averaged results are finally used for the quantitative evaluation.

We highlight the effectiveness of the proposed methods ($l_2$-CoSpace and $l_1$-CoSpace) in the CML-based task by quantitatively comparing with several state-of-the-art baselines, including the original MS-LIDAR data (Baseline), PCA-based joint dimensionality reduction (P-JDR) [2], locality preserving projection (LPP)-based unsupervised MA (L-USMA) [15], and LPP-based supervised MA (L-SMA) [16].

For a fair comparison, we aim at maximizing the classification performance by selecting the optimal parameters of different algorithms using 10-fold cross-validation (CV) on the training data. The parameters are feature dimension ($d$), regularization parameters ($\alpha, \beta$) in CoSpace, and the number of nearest neighbors ($k$) as well as the standard deviation of Gaussian kernel function ($\sigma$). More specifically, we select the parameters $d$ and $k$ ranging from {10, 20, ..., 50}, while the optimal $\sigma$, $\alpha$, and $\beta$ can be found from {10⁻², 10⁻<fnref n="1" />, 10⁰, 10¹, 10²}. Please note that the CV is a

widely used and very effective strategy in the machine learning community to determine the model parameters, as long as the training samples with labels are given.

### C. Results and Analysis

The classification maps of different algorithms using LSVM and CCF classifiers are visualized in Fig. 3, and correspondingly, the quantitative results for those compared methods in terms of OA, AA, and $\kappa$ are listed in Table I, where the parameters are experimentally determined by the 10-fold CV on the training set.

The baseline yields poor classification performance, due to the limitation of the feature representation ability. By jointly embedding MS-Lidar and HS data, P-JDR tends to obtain a higher classification accuracy than the baseline, particularly using CCF classifier (nearly 3% improvement). Due to fully considering the local topological structure of the input, L-USMA performs better than baseline and P-JDR. Similarly, L-SMA constructs a Linear Discriminant Analysis (LDA)-like graph based on the available labels, achieving better performance than L-USMA. Different with MA-based approaches (e.g., L-USMA, L-SMA) that attempt to directly find an aligned latent space from different modalities, the CoSpace-based models aim at jointly learning a latent subspace and a

<footnote n="1">¹To meet the CML's problem setting and investigate the transferability of the proposed method, the training samples are collected in a limited region.</footnote>
<table><caption>TABLE I
QUANTITATIVE PERFORMANCE COMPARISONS OF DIFFERENT ALGORITHMS IN TERMS OF OA, AA, AND $κ$. THE BEST ONE IS SHOWN IN BOLD</caption><thead><tr><th>Algorithm</th><th colspan="2">Baseline (%)</th><th colspan="2">P-JDR (%)</th><th colspan="2">L-USMA (%)</th><th colspan="2">L-SMA (%)</th><th colspan="2">ℓ<sub>2</sub>-CoSpace (%)</th><th colspan="2">ℓ<sub>1</sub>-CoSpace (%)</th></tr><tr><th>Parameter (d, k, σ, α, β)</th><th colspan="2">(9, -, -, -, -)</th><th colspan="2">(20, -, -, -, -)</th><th colspan="2">(30, 10, 1, -, -)</th><th colspan="2">(20, -, -, -, -)</th><th colspan="2">(30, -, -, 0.1, 0.1)</th><th colspan="2">(30, -, -, 0.1, 0.01)</th></tr><tr><th>Classifier</th><th>LSVM</th><th>CCF</th><th>LSVM</th><th>CCF</th><th>LSVM</th><th>CCF</th><th>LSVM</th><th>CCF</th><th>LSVM</th><th>CCF</th><th>LSVM</th><th>CCF</th></tr></thead><tbody><tr><td>OA</td><td>53.77</td><td>59.79</td><td>52.60</td><td>62.59</td><td>53.21</td><td>62.46</td><td>54.76</td><td>62.76</td><td>57.73</td><td>66.25</td><td>58.15</td><td><strong>68.17</strong></td></tr><tr><td>AA</td><td>48.26</td><td>47.94</td><td>47.17</td><td>53.91</td><td>49.03</td><td>54.00</td><td>52.67</td><td>56.45</td><td>53.17</td><td>60.78</td><td>56.43</td><td><strong>62.28</strong></td></tr><tr><td>κ</td><td>0.4531</td><td>0.5094</td><td>0.4408</td><td>0.5426</td><td>0.4478</td><td>0.5420</td><td>0.4661</td><td>0.5481</td><td>0.4931</td><td>0.5892</td><td>0.5007</td><td><strong>0.6093</strong></td></tr><tr><td>Class1</td><td>77.16</td><td>57.75</td><td>78.62</td><td>65.75</td><td>81.72</td><td>69.03</td><td>80.34</td><td>72.07</td><td><strong>83.41</strong></td><td>71.60</td><td>80.97</td><td>76.95</td></tr><tr><td>Class2</td><td>45.62</td><td>52.65</td><td>41.66</td><td>81.88</td><td>44.56</td><td>67.88</td><td>50.85</td><td>76.21</td><td><strong>87.16</strong></td><td>61.42</td><td>47.62</td><td>85.79</td></tr><tr><td>Class3</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>94.17</td><td>94.17</td></tr><tr><td>Class4</td><td>87.56</td><td>91.12</td><td>88.33</td><td>88.70</td><td>91.84</td><td>91.54</td><td><strong>95.19</strong></td><td>92.44</td><td>95.09</td><td>93.25</td><td>93.67</td><td>91.83</td></tr><tr><td>Class5</td><td>68.00</td><td>48.13</td><td>59.01</td><td>63.41</td><td>62.28</td><td>71.88</td><td>85.93</td><td>79.75</td><td><strong>89.56</strong></td><td>81.69</td><td>88.17</td><td>85.02</td></tr><tr><td>Class6</td><td>7.47</td><td>0.00</td><td>11.11</td><td>0.76</td><td>2.12</td><td>0.10</td><td><strong>15.46</strong></td><td>0.00</td><td>2.94</td><td>0.00</td><td>1.18</td><td>0.00</td></tr><tr><td>Class7</td><td>23.50</td><td>0.00</td><td>0.55</td><td>21.31</td><td>40.44</td><td>12.57</td><td>42.62</td><td>39.89</td><td>41.53</td><td>45.90</td><td><strong>54.64</strong></td><td>48.63</td></tr><tr><td>Class8</td><td>79.91</td><td>72.78</td><td>78.89</td><td>67.79</td><td>76.99</td><td>71.86</td><td>75.08</td><td>74.46</td><td>72.46</td><td><strong>83.78</strong></td><td>79.02</td><td>80.27</td></tr><tr><td>Class9</td><td>67.20</td><td>77.07</td><td>65.53</td><td>77.50</td><td>65.38</td><td>77.14</td><td>65.90</td><td>74.97</td><td>72.37</td><td>77.87</td><td>69.97</td><td><strong>80.57</strong></td></tr><tr><td>Class10</td><td>11.24</td><td><strong>31.17</strong></td><td>14.81</td><td>28.90</td><td>8.70</td><td>28.17</td><td>21.78</td><td>29.81</td><td>21.29</td><td>25.83</td><td>19.09</td><td><strong>31.17</strong></td></tr><tr><td>Class11</td><td>26.58</td><td>41.45</td><td>24.94</td><td>45.19</td><td>24.66</td><td>49.37</td><td>28.81</td><td>47.20</td><td>29.34</td><td>52.85</td><td>44.25</td><td><strong>56.63</strong></td></tr><tr><td>Class12</td><td>31.73</td><td>35.94</td><td>33.63</td><td><strong>40.54</strong></td><td>31.43</td><td>36.94</td><td>27.63</td><td>36.04</td><td>21.32</td><td>35.44</td><td>36.14</td><td>36.64</td></tr><tr><td>Class13</td><td>31.10</td><td>33.26</td><td>28.82</td><td>34.58</td><td>39.63</td><td>36.87</td><td>27.19</td><td>38.75</td><td>20.72</td><td>39.10</td><td>31.42</td><td><strong>44.25</strong></td></tr><tr><td>Class14</td><td>46.90</td><td>30.49</td><td>41.60</td><td>30.10</td><td>48.08</td><td>33.16</td><td>48.03</td><td>45.74</td><td><strong>53.85</strong></td><td>47.71</td><td>52.49</td><td>50.05</td></tr><tr><td>Class15</td><td>58.66</td><td>43.43</td><td>59.55</td><td>40.60</td><td><strong>60.25</strong></td><td>40.78</td><td>57.42</td><td>38.40</td><td>55.57</td><td>53.04</td><td>54.96</td><td>38.71</td></tr><tr><td>Class16</td><td>26.16</td><td>20.81</td><td>16.18</td><td>32.02</td><td>17.80</td><td>27.63</td><td>28.20</td><td>21.70</td><td>46.45</td><td>28.60</td><td><strong>37.80</strong></td><td>31.59</td></tr><tr><td>Class17</td><td>0.00</td><td>7.38</td><td>0.00</td><td>9.40</td><td>0.00</td><td>10.74</td><td>6.04</td><td>16.78</td><td>2.01</td><td>39.60</td><td>23.49</td><td><strong>59.06</strong></td></tr><tr><td>Class18</td><td>55.69</td><td>48.03</td><td>68.91</td><td>78.77</td><td>60.09</td><td>80.11</td><td>71.82</td><td>82.63</td><td>67.39</td><td>86.67</td><td>84.90</td><td><strong>91.28</strong></td></tr><tr><td>Class19</td><td>64.32</td><td><strong>88.09</strong></td><td>76.49</td><td>88.35</td><td>68.10</td><td>87.05</td><td>73.75</td><td>83.88</td><td>71.36</td><td>87.14</td><td>85.05</td><td>88.01</td></tr><tr><td>Class20</td><td>56.46</td><td>79.18</td><td>54.77</td><td>82.70</td><td>56.53</td><td><strong>86.36</strong></td><td>51.31</td><td>78.33</td><td>55.35</td><td>78.39</td><td>49.54</td><td>75.00</td></tr></tbody></table>

regression matrix bridging the learned subspace with labels. This might make the learned features more discriminative, thereby yielding the best classification performance. Remarkably still, we found that there is a further improvement in $ℓ_1$-CoSpace over $ℓ_2$-CoSpace. The possible reason for that is the use of sparse regression matrix, which is easy to implement the sparse-promoting structural learning.

With LSVM classifier, $ℓ_1$-CoSpace improves the OAs of 4.35%, 5.55%, 4.94%, 3.39%, and 0.42%, respectively, compared to baseline, P-JDR, L-USMA, L-SMA, and even $ℓ_2$-CoSpace. While using CCF classifier, $ℓ_1$-CoSpace obviously increases by 8.38%, 5.58%, 5.71%, 5.41%, and 1.92% for the above five methods in terms of OA. As expected, the similar trends in AA and $κ$ can be also observed in Table I.

IV. CONCLUSION

In this letter, we investigate a CML-related problem by using the heterogeneous RS data (MS-Lidar and HS data). Concretely, we propose a novel joint sparse subspace learning ($ℓ_1$-CoSpace) model, which is an improved version of CoSpace, by simultaneously learning a shared feature subspace and a sparse regression matrix. Benefiting from sparse modeling, the proposed $ℓ_1$-CoSpace can interpret and mine the intrinsic structure of the data more effectively, resulting in a further performance improvement.

REFERENCES

[1] M. D. Mura, S. Prasad, F. Pacifici, P. Gamba, J. Chanussot, and J. A. Benediktsson, “Challenges and opportunities of multimodality and data fusion in remote sensing,” Proc. IEEE, vol. 103, no. 9, pp. 1585–1601, Sep. 2015.

[2] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, “Cospace: Common subspace learning from hyperspectral-multispectral correspondences,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 7, pp. 4349–4359, Jul. 2019.

[3] N. Yokoya et al., “Open data for global multimodal land use classification: Outcome of the 2017 IEEE GRSS data fusion contest,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 11, no. 5, pp. 1363–1377, May 2018.

[4] B. Rasti, P. Ghamisi, and R. Gloaguen, “Hyperspectral and LiDAR fusion using extinction profiles and total variation component analysis,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 7, pp. 3997–4007, Jul. 2017.

[5] D. Hong, N. Yokoya, and X. X. Zhu, “Learning a robust local manifold representation for hyperspectral dimensionality reduction,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 10, no. 6, pp. 2960–2975, Jun. 2017.

[6] D. Tuia, M. Volpi, M. Trolliet, and G. Camps-Valls, “Semisupervised manifold alignment of multimodal remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 12, pp. 7708–7720, Dec. 2014.

[7] B. Banerjee and K. M. Buddhiraju, “Domain adaptation in the absence of source domain labeled samples—A coclustering-based approach,” IEEE Geosci. Remote Sens. Lett., vol. 13, no. 12, pp. 1905–1909, Dec. 2016.

[8] B. Du, S. Wang, C. Xu, N. Wang, L. Zhang, and D. Tao, “Multi-task learning for blind source separation,” IEEE Trans. Image Process., vol. 27, no. 9, pp. 4219–4231, Sep. 2018.

[9] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, “An augmented linear mixing model to address spectral variability for hyperspectral unmixing,” IEEE Trans. Image Process., vol. 28, no. 4, pp. 1923–1938, Apr. 2019.

[10] D. Cai, X. He, and J. Han, “Spectral regression: A unified approach for sparse subspace learning,” in Proc. 7th IEEE Int. Conf. Data Mining (ICDM), Oct. 2007, pp. 73–82.

[11] D. Hong and X. X. Zhu, “SULoRA: Subspace unmixing with low-rank attribute embedding for hyperspectral data analysis,” IEEE J. Sel. Top. Signal Process., vol. 12, no. 6, pp. 1351–1363, Dec. 2018.

[12] T. Rainforth and F. Wood, “Canonical correlation forests,” 2015, arXiv:1507.05444. [Online]. Available: https://arxiv.org/abs/1507.05444

[13] B. Le Saux, N. Yokoya, R. Hansch, and S. Prasad, “2018 IEEE GRSS data fusion contest: Multimodal land use classification [technical committees],” IEEE Geosci. Remote Sens. Mag., vol. 6, no. 1, pp. 52–54, Mar. 2018.

[14] D. Hong, W. Liu, J. Su, Z. Pan, and G. Wang, “A novel hierarchical approach for multispectral palmprint recognition,” Neurocomputing, vol. 151, pp. 511–521, Mar. 2015.

[15] X. He and P. Niyogi, “Locality preserving projections,” in Proc. NIPS, Dec. 2004, pp. 153–160.

[16] D. Hong, N. Yokoya, N. Ge, J. Chanussot, and X. X. Zhu, “Learnable manifold alignment (LeMA): A semi-supervised cross-modality learning framework for land cover and land use classification,” ISPRS J. Photogram. Remote Sens., vol. 147, pp. 193–205, Jan. 2019.