# SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot

► To cite this version:

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, et al. SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60, pp.5518615. 10.1109/TGRS.2021.3130716. hal-03932889

HAL Id: hal-03932889

https://hal.science/hal-03932889v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers

Danfeng Hong, *Senior Member, IEEE*, Zhu Han, *Student Member, IEEE*, Jing Yao, Lianru Gao, *Senior Member, IEEE*, Bing Zhang, *Fellow, IEEE*, Antonio Plaza, *Fellow, IEEE*, and Jocelyn Chanussot, *Fellow, IEEE*

**Abstract**—Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called SpectralFormer. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse “soft” residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixel- and patch-wise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

**Index Terms**—Hyperspectral image classification, convolutional neural networks, deep learning, local contextual information, remote sensing, sequence data, skip fusion, transformer.

## I. INTRODUCTION

IN hyperspectral (HS) imaging, hundreds of (narrow) wavelength bands are collected at each pixel across the complete

This work was supported in part by the National Natural Science Foundation of China under Grant 42030111 and Grant 41722108, and by the MIAI@Grenoble Alpes (ANR-19-P3IA-0003) and the AXA Research Fund. (Corresponding author: Lianru Gao)

D. Hong, J. Yao and L. Gao are with the Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, 100094 Beijing, China. (e-mail: hongdf@aircas.ac.cn; yaojing@aircas.ac.cn; gaolr@aircas.ac.cn)

Z. Han and B. Zhang are with the Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China, and also with the College of Resources and Environment, University of Chinese Academy of Sciences, Beijing 100049, China. (e-mail: hanzhu19@mails.ucas.ac.cn; zb@radi.ac.cn)

A. Plaza is with the Hyperspectral Computing Laboratory, Department of Technology of Computers and Communications, Escuela Politécnica, University of Extremadura, 10003 Cáceres, Spain. (e-mail: aplaza@unex.es).

J. Chanussot is with the Univ. Grenoble Alpes, INRIA, CNRS, Grenoble INP, LJK, F-38000 Grenoble, France, and also with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China. (e-mail: jocelyn@hi.is)

electromagnetic spectrum, which enables the identification or detection of materials at a fine-grained level, particularly for those that have extremely similar spectral signatures in visual cues (e.g., RGB) [1]. This provides a great potential in a variety of high-level Earth observation (EO) missions, such as accurate land cover mapping, precision agriculture, target / object detection, urban planning, tree species classification, mineral exploration, and so on.

A general sequential process in a HS image classification system consists of image restoration (e.g., denoising, missing data recovery) [2]–[5], dimensionality reduction [6], [7], spectral unmixing [8]–[12], and feature extraction [13]–[16]. Among them, feature extraction is a crucial step in HS image classification, which has received increasing attention by researchers. Over the past decade, a large number of advanced hand-crafted and subspace learning-based feature extraction approaches have been proposed for HS image classification [17]. These methods are capable of performing well in small-sample classification problems. However, they tend to meet the performance bottleneck, when the training size gradually increases and the training set becomes more complex. The possible reason is due to the limited data fitting and representation ability of these traditional methods. Inspired by the great success of deep learning (DL), that is capable of finding out connotative, intrinsic, and potentially valuable knowledge from the vast amounts of pluralistic data [18], enormous efforts have been made in designing and adding advanced modules in networks to extract more diagnostic features from remote sensing data. For example, Zhao et al. [19] developed a joint classification framework using HS and light detection and ranging (LiDAR) data, which has been shown to be excellent at extracting features from multisource RS data. Zhang et al. [20] designed an extraordinary patch-to-patch convolutional neural network (CNN), which has yielded significantly better results than other techniques.

In recent years, many well-recognized backbone networks have been widely and successfully applied in the HS image classification task [28], such as autoencoders (AEs), CNNs, recurrent neural networks (RNNs), generative adversarial networks (GANs), capsule networks (CapsNet), graph convolutional networks (GCNs). Chen et al. [21] stacked multiple autoencoder networks to extract deep feature representations from dimension-reduced HS images – generated by principal component analysis (PCA) [29] – with the application to HS image classification. Chen et al. [22] employed CNNs instead of stacked AEs to semantically extract spatial-spectral features by considering local contextual information of HS images,
<figure><img src="image_2.png" /><figcaption>Fig. 1. An overview of currently well-recognized backbone networks for the HS image classification task, such as autoencoders [21], CNNs [22], RNNs [23], GANs [24], CapsNet [25], GCNs [26], and Transformers [27].</figcaption></figure>

achieving higher classification accuracies. Hang et al. [23] designed a cascaded RNN for HS image classification by taking advantage of RNNs that can model the sequentiality to represent the relations of neighboring spectral bands effectively. In [24], GANs were improved, making them applicable to the HS image classification task, with the input of three PCA components and random noises. Paoletti et al. [25] extended a CNN-based model by defining a novel spatial-spectral capsule unit, yielding a high-performance classification framework of HS images and meanwhile reducing the complexity of the network design. Hong et al. [26] made a comprehensive comparison between CNNs and GCNs on the classification of HS images both qualitatively and quantitatively, and proposed a mini-batch GCN (miniGCNs), providing a feasible solution for addressing the issue of large graphs in GCNs, for state-of-the-art HS image classification.

Although these backbone networks and their variants have been able to obtain promising classification results, their ability in characterizing spectral series information (particularly in capturing subtle spectral discrepancies along spectral dimension) remains inadequate. Fig. 1 gives an overview illustration of these state-of-the-art backbone networks in the HS image classification task. The specific limitations can be roughly summarized as follows.

* As a mainstream backbone architecture, CNNs have shown their powerful ability in extracting spatially structural information and locally contextual information from HS images. Nevertheless, on the one hand, CNNs can hardly be capable of capturing the sequence attributes well, particularly middle- and long-term dependencies. This unavoidably meets the performance bottleneck in the HS image classification task, especially when the to-be-classified categories are of a great variety and extremely similar in spectral signatures. On the other hand, CNNs are overly concerned with spatial content information, which spectrally distorts sequential information in the learned features. This, to great extent, enables more difficulties of mining diagnostic spectral attributes.

* Unlike CNNs, RNNs are designed for sequence data, which accumulatively learns spectral features band-by-band from HS images in an orderly fashion. Such a mode extremely depends on the order of spectral bands and tends to generate gradient vanishing, thereby being

hard to learn long-term dependencies [30]. This might further lead to the difficulty of capturing spectrally salient changes in time series. More importantly, there are usually tons of HS samples (or pixels) available in real HS image scenes, yet RNNs fail to train the models in parallel, limiting the classification performance in practical applications.

* For other backbone networks, i.e., GANs, CapsNet, GCNs, despite their respective advantages in learning spectral representations (e.g., robustness, equivalence, long-range relevance between samples), one thing in common is that almost all of them could be inherently incompetent for modeling sequential information effectively. That is, the utilization of spectral information is insufficient (being a critical bottleneck in fine land cover classification or mapping using HS data).

Being aimed at the aforementioned limitations, we rethink the HS image classification process from a sequence data perspective with current state-of-the-art transformers [27]. Totally different from CNNs and RNNs, transformers are (at present) one of the cutting-edge backbone networks, owing to the use of self-attention techniques, which are well-designed for the sake of processing and analyzing sequential (or time series) data more effectively. This will provide a good fit for HS data processing and analysis, e.g., HS image classification. It is well-known that the self-attention block in transformers enables to capture globally sequential information by the means of positional encoding [31]. However, there also exist some drawbacks in transformers that hinder its performance to be further improved. For example,

1) although transformers perform outstandingly in solving the problem of long-term dependencies of spectral signatures, they lose the power of capturing locally contextual or semantic characteristics (cf. CNNs or RNNs);

2) as mentioned in [32], the skip connection plays a crucial role in transformers. This might be explained well by either using “residuals” to make the gradients better propagated or enhancing “memories” to reduce the forgetting or loss of key information. But unfortunately, the simple additive skip connection operation only occurs within each transformer block, weakening the connectivity across different layers or blocks.

For these reasons, we aim to develop a novel transformers-
based network architecture, SpectralFormer for short, enabling the high-performance HS image classification task. SpectralFormer provides point-to-point solutions corresponding to the above two drawbacks. More specifically, SpectralFormer is capable of learning locally spectral representations from multiple neighboring bands rather than single band (in original transformers) in each encoded position, e.g., group-wise versus band-wise embeddings. Furthermore, a cross-layer skip connection is designed in SpectralFormer to progressively convey memory-like components from shallow to deep layers by adaptively learning to fuse their “soft” residuals. The main contributions of this paper can be summarized as follows.

* We revisit the HS image classification problem from a sequential perspective and propose a new transformer-based backbone network, called SpectralFormer, in order to substitute for CNNs or RNNs-based architectures. To the best of our knowledge, this is the first time that transformers (without any pre-processing operation, e.g., feature extraction using convolution and recurrent units or other transformation techniques) are purely applied to the HS image classification task.

* We devise two simple but effective modules in SpectralFormer, i.e., group-wise spectral embedding (GSE) and cross-layer adaptive fusion (CAF), to learn locally detailed spectral representations and convey memory-like components from shallow to deep layers, respectively.

* We qualitatively and quantitatively evaluate the classification performance of the proposed SpectralFormer on three representative HS datasets, i.e., Indian Pines, Pavia University, and University of Houston, with extensive ablation studies. The experimental results demonstrate a significant superiority over classic transformers (with an increase of approximately 10% OA) and other state-of-the-art backbone networks (at least 2% OA improvement).

The remaining of the paper is organized as follows. Section II first reviews classic transformers-related literature and then details the proposed SpectralFormer with two well-designed modules for HS image classification. Extensive experiments are conducted with ablation studies and discussions in Section III. Section IV draws comprehensive conclusions and a brief outlook on future possible research directions.

## II. SPECTRALFORMER

In this section, we start to review some preliminaries of classic transformers. On this basis, we propose the SpectralFormer with two well-designed modules, i.e., GSE and CAF, making it more applicable to the HS image classification task. Finally, we also investigate the ability that the proposed SpectralFormer models the spatially contextual information with the input of image patches.

### A. A Brief Review of Transformers

As is known to all, transformers [27] have cut a conspicuous figure in processing sequence-to-sequence problems in natural language processing (NLP), e.g., machine translation. Since they abandon the sequence dependence characteristic in

<figure><img src="image_3.png" /><figcaption>Fig. 2. An illustration of the attention mechanism in transformers. (a) Self-attention module; (b) Multi-head attention.</figcaption></figure>

RNNs and alternatively introduce a brand-new self-attention mechanism. This enables the global information (long-term dependencies) capture for the units at any position, which greatly promotes the development of time series data processing models. Even not only limiting in NLP, image processing and computer vision fields have also begun exploring the transformer architecture. Very recently, the vision transformer (ViT) [33] seems to have achieved or approached CNNs-based state-of-the-art effects on various vision domain tasks, providing new insight, inspiration, and creative space on vision-related tasks.

The success of transformers, to a great extent, depends on the use of multi-head attention, where multiple self-attention (SA) [34] layers are stacked and integrated. As the name suggests, SA mechanism is better at capturing the internal correlation of data or features, thereby reducing the dependence on external information. Fig. 2(a) illustrates the process of the SA module in transformers. More specifically, the SA layer can be performed according to the following six steps:

**Step 1.** Input the sequence data $x$ with the length of $m$, where $x_i$, $i = 1, ..., m$ denotes either a scalar or a vector.

**Step 2.** The feature embedding, denoted as $a_i$, is obtained on each scalar or vector $x_i$ by a shared matrix $W$.

**Step 3.** Each embedding multiplies by three different transformation matrices $W_q$, $W_k$, $W_v$, respectively, yielding three vectors, i.e., Query ($Q = [q_1, ..., q_m]$), Key ($K = [k_1, ..., k_m]$), Value ($V = [v_1, ..., v_m]$).

**Step 4.** Compute the attention score $s$ between each $Q$ vector and each $K$ vector in the form of inner product, e.g., $q_i \cdot k_j$, and to stabilize the gradients, the scaled score is obtained by normalization, i.e., $s_{i,j} = q_i \cdot k_j / \sqrt{d}$, where $d$ is the dimension of $q_i$ or $k_j$.

**Step 5.** The Softmax activation function is performed on $s$; we then have in the position-1 for instance: $\hat{s}_{1,i} = e^{s_{1,i}} / \sum_j e^{s_{1,j}}$.

**Step 6.** Generate attention representations $z = [z_1, ..., z_m]$, e.g., $z_1 = \sum_i \hat{s}_{1,i}v_i$.

To sum up, the SA layer can be integrally formulated as follows

$$z = \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V. \quad (1)$$
<figure><img src="image_4.png" /><figcaption>Fig. 3. An overview illustration of the proposed SpectralFormer network for the HS image classification task. SpectralFormer consists of two well-designed modules, i.e., GSE and CAF, making it (the transformers-based backbone) better applicable to HS images.</figcaption></figure>

With Eq. (1), multiple different SA layers can be assembled to be a multi-head attention, as shown in Fig. 2(b). In detail, we first obtain multiple attention representations (e.g., $h = 8$), denoted as $z^h$, $h = 1, ..., 8$, and concatenate them to be a larger feature matrix. A linear transformation matrix (e.g., $W_o$) is finally used to make the feature dimension identical to the input data.

It should be noted, however, that there is no position information in the SA layer, which fails to make use of sequence information. For this reason, the position information is encoded into the feature embedding. The embedding with time signal can be thus formulated as $a_i + e_i$, where $e_i$ denotes a unique positional vector given manually.

## B. Overview of SpectralFormer

We aim at developing a novel and generic ViT-based baseline network (i.e., SpectralFormer) with a focus on the spectrometric characteristics, making it well-applicable to the highly accurate and fine classification of HS images. To this end, we devise two key modules, i.e., GSE and CAF, and integrate them into the transformer framework, in order to improve the detail-capturing capacity of subtle spectral discrepancies and enhance the information transitivity (or connectivity) between layers (i.e., reduce the information loss with the gradual deepening of layers), respectively. Moreover, the proposed SpectralFormer is not only applied to the pixel-wise HS image classification, but also extensible to the spatial-spectral classification with the batch-wise input, yielding the spatial-spectral SpectralFormer version. Fig. 3 illustrates an overview of the proposed SpectralFormer in the HS image classification task, while Table I details the definition of notations used in the proposed SpectralFormer.

## C. Group-wise Spectral Embedding (GSE)

Unlike the discrete sequentiality in classic transformers or ViT (e.g., image patches), hundreds or thousands of spectral channels in HS images are densely sampled from the electromagnetic spectrum at a subtle interval (e.g., 10nm), yielding approximately continuous spectral signatures. The spectral information in different position reflects different absorption characteristics corresponding to different wavelength. This to a great extent shows the physical properties of the current material. Capturing locally detailed absorption (or changes) of such spectral signatures is a crucial factor to accurately and finely classify the materials lying in the HS scene. For this purpose, we propose to learn group-wise spectral embeddings, i.e., GSE, instead of band-wise input and representations. Given a spectral signature (a pixel in the HS image) $x = [x_1, x_2, ..., x_m] \in \mathbb{R}^{1 \times m}$, the feature embeddings $a$ obtained by classic transformers are formulated by

$$A = wx, \qquad (2)$$

where $w \in \mathbb{R}^{d \times 1}$ denotes the linear transformation that is equivalently used for all bands in spectral signatures and $A \in \mathbb{R}^{d \times m}$ collects the output features. Whereas the proposed GSE learns the feature embeddings from locally spectral profiles (or neighboring bands). Thus, the GSE can be modeled as

$$\dot{A} = W X = W g(x), \qquad (3)$$

where $W \in \mathbb{R}^{d \times n}$ and $X \in \mathbb{R}^{n \times m}$ correspond to the grouped representations with respect to the variables $w$ and $x$, respectively, $n$ represents the number of neighboring bands. The variable $W$ can be simply seen as one layer of network, which can be optimized by updating the whole network. The function $g(\cdot)$ denotes the overlapping grouping operation in regard to the variable $x$, i.e.,

$$X = g(x) = [x_1, ..., x_q, ..., x_m], \qquad (4)$$
<table><caption>TABLE I
DEFINITION OF NOTATIONS USED IN THE PROPOSED SPECTRALFORMER</caption><thead><tr><th>Notation</th><th>Definition</th><th>Type</th><th>Size</th></tr></thead><tbody><tr><td><i>m</i></td><td>the number of spectral bands</td><td>scalar</td><td>1 × 1</td></tr><tr><td><i>x</i></td><td>spectral signature of a HS pixel</td><td>vector</td><td>1 × <i>m</i></td></tr><tr><td><i>x</i><sub>i</sub></td><td>reflectance or radiance value in the <i>i</i>-th band location</td><td>scalar</td><td>1 × 1</td></tr><tr><td><i>d</i></td><td>the dimension of the feature embeddings</td><td>scalar</td><td>1 × 1</td></tr><tr><td><i>w</i></td><td>the linear transformation w.r.t a pixel (sample)</td><td>vector</td><td><i>d</i> × 1</td></tr><tr><td><b>A</b></td><td>the feature embeddings</td><td>matrix</td><td><i>d</i> × <i>m</i></td></tr><tr><td><i>n</i></td><td>the number of considered HS pixels</td><td>scalar</td><td>1 × 1</td></tr><tr><td><b>W</b></td><td>the linear transformations w.r.t. <i>n</i> pixels (samples)</td><td>matrix</td><td><i>d</i> × <i>n</i></td></tr><tr><td><b>X</b></td><td>spectral signatures of <i>n</i> pixels</td><td>matrix</td><td><i>n</i> × <i>m</i></td></tr><tr><td><i>g</i>(·)</td><td>the overlapping grouping operation w.r.t. the variable (·)</td><td>—</td><td>—</td></tr><tr><td><b>A&#x0302;</b></td><td>the feature embeddings of GSE</td><td>matrix</td><td><i>d</i> × <i>m</i></td></tr><tr><td>[·]</td><td>the round operation</td><td>—</td><td>—</td></tr><tr><td><b>z</b><sup>(<i>l</i>)</sup></td><td>the feature representations in the (<i>l</i>)-th layer</td><td>vector</td><td>1 × <i>d</i><sub><i>z</i></sub></td></tr><tr><td><b>ŵ</b></td><td>the learnable network parameter for adaptive fusion</td><td>vector</td><td>1 × 2</td></tr><tr><td><b>z&#x0302;</b><sup>(<i>l</i>)</sup></td><td>the fused representations in the (<i>l</i>)-th layer with the CAF</td><td>vector</td><td>1 × <i>d</i><sub><i>z</i></sub></td></tr></tbody></table>

where $\mathbf{x}^q = [x_{q-\lfloor \frac{n}{2} \rfloor}, \dots, x_q, \dots, x_{q+\lfloor \frac{n}{2} \rfloor}]^\top \in \mathbb{R}^{n \times 1}, \lfloor \bullet \rfloor$ denotes the round operation. Fig. 4 illustrates the differences between band-wise and group-wise spectral embeddings in transformers-based backbone networks, i.e., BSE versus GSE.

### D. Cross-layer Adaptive Fusion (CAF)

The skip connection (SC) mechanism has been proven to be an effective strategy in deep networks, which can enhance information exchange between layers and reduce information loss in the network learning process. The use of SC has recently gained tremendous success in image recognition and segmentation, e.g., short SC for ResNet [35] and long SC for U-Net [36]. It should be noted, however, that the information “memory” ability of the short SC remains limited, while the long SC tends to yield insufficient fusion due to a big gap between high- and low-level features. This is also a key problem existed in transformers, which will pose a new challenge to the architecture design of transformers. To this end, we devise a middle-range SC in SpectralFormer to adaptively learn cross-layer feature fusion (i.e., CAF, see Fig. 5).

Let $\mathbf{z}^{(l-2)} \in \mathbb{R}^{1 \times d_z}$ and $\mathbf{z}^{(l)} \in \mathbb{R}^{1 \times d_z}$ be the outputs (or representations) in the $(l-2)$-th and $(l)$-th layers, respectively, CAF can be then expressed by

$$ \hat{\mathbf{z}}^{(l)} \leftarrow \ddot{\mathbf{w}} \begin{bmatrix} \mathbf{z}^{(l)} \\ \mathbf{z}^{(l-2)} \end{bmatrix}, \quad (5) $$

where $\hat{\mathbf{z}}^{(l)}$ denotes the fused representations in the $(l)$-th layer with the proposed CAF, and $\ddot{\mathbf{w}} \in \mathbb{R}^{1 \times 2}$ is the learnable network parameter for adaptive fusion. It should be noted that our CAF only skips one encoder, e.g., from $\mathbf{z}^{(l-2)}$ (the output of Encoder 1) to $\mathbf{z}^{(l)}$ (the output of Encoder 3) in Fig. 5. The reasons for this setting are two-fold. On the one hand, there is a big semantic gap between low- and deep-level features obtained from the shallow and deep layers, respectively. If the relatively long SC, e.g., two, three, and even more encoders, is used, this then might lead to an insufficient fusion and potential information loss. On the other hand, the HS image classification can be often regarded as a small-sample problem, due to the limited available training (need

<figure><img src="image_5.png" /><figcaption>Fig. 4. An illustrative comparison of band-wise and group-wise spectral embeddings (BSE versus GSE) in transformers.</figcaption></figure>

to be labeled manually) samples. A “small” or “shallow” backbone network, e.g., 4 or 5 layers, may already be a good fit for the HS image classification task. As a result, this, to some extent, can explain why we propose to skip only one encoder in the CAF module (since a 4 or 5-layered shallow network is small, which is not capable of adding multiple CAF modules).

### E. Spatial-Spectral SpectralFormer

Beyond the pixel-wise HS image classification, we similarly investigate the patch-wise input (inspired by CNNs), yielding the spatial-spectral SpectralFormer version, i.e., patch-wise SpectralFormer. Different from CNNs, that directly input a 3-D patch cube, we unfold the 2-D patch of each band to the corresponding 1-D vector representations. Given a 3-D cube $\mathcal{X} \in \mathbb{R}^{m \times w \times h}$ ($w$ and $h$ are the width and length of the patch),
<figure><img src="image_6.png" /><figcaption>Fig. 5. A diagram of cross-layer adaptive fusion (CAF) module in the proposed SpectralFormer.</figcaption></figure>

which can be unfolded along with the spatial direction, we then have

$$ \hat{\mathbf{X}} = [\vec{\mathbf{x}}_1, \dots, \vec{\mathbf{x}}_i, \dots, \vec{\mathbf{x}}_m], \quad (6) $$

where $\vec{\mathbf{x}}_i \in \mathbb{R}^{wh \times 1}$ denotes the unfolded patch for the i-th band. Such an input way can to a great extent preserve the spectrally sequential information in network learning and meanwhile consider spatially contextual information.

### III. EXPERIMENTS

In this section, three well-known HS datasets are firstly described, the implementation details and compared state-of-the-art methods are then introduced. Finally, extensive experiments are conducted with ablation analysis to assess the HS image classification performance of the proposed SpectralFormer both quantitatively and qualitatively.

#### A. Data Description

1) **Indian Pines Data:** The first HS data was collected in 1992 by using the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor over North-Western Indiana, USA. The HS image consists of 145 × 145 pixels at a ground sampling distance (GSD) of 20m, and 220 spectral bands covering the wavelength range of 400nm to 2500nm with a 10m spectral resolution. After removing 20 noisy and water absorption bands, 200 spectral bands are retained, i.e., 1-103, 109-149, 164-219. There are 16 mainly-investigated categories in this studied scene. The class name and the number of samples used for training and testing in the classification task are listed in Table II, while the spatial distribution of training and testing sets are also given in Fig. 7 to reproduce the classification results.

2) **Pavia University Data:** The second HS scene was acquired by the Reflective Optics System Imaging Spectrometer (ROSIS) sensor over Pavia University and its surroundings, Pavia, Italy. The sensor can capture 103 spectral bands ranging from 430nm to 860nm, and the image consists of 610 × 340 pixels at a GSD of 1.3m. This scene includes 9 land cover classes, where the fixed number of training and testing samples are detailed in Table III and spatially visualized in Fig. 8.

<table><caption>TABLE II
LAND-COVER CLASSES OF THE INDIAN PINES DATASET, WITH THE STANDARD TRAINING AND TESTING SETS FOR EACH CLASS.</caption><thead><tr><th>Class No.</th><th>Class Name</th><th>Training</th><th>Testing</th></tr></thead><tbody><tr><td>1</td><td>Corn Notill</td><td>50</td><td>1384</td></tr><tr><td>2</td><td>Corn Mintill</td><td>50</td><td>784</td></tr><tr><td>3</td><td>Corn</td><td>50</td><td>184</td></tr><tr><td>4</td><td>Grass Pasture</td><td>50</td><td>447</td></tr><tr><td>5</td><td>Grass Trees</td><td>50</td><td>697</td></tr><tr><td>6</td><td>Hay Windrowed</td><td>50</td><td>439</td></tr><tr><td>7</td><td>Soybean Notill</td><td>50</td><td>918</td></tr><tr><td>8</td><td>Soybean Mintill</td><td>50</td><td>2418</td></tr><tr><td>9</td><td>Soybean Clean</td><td>50</td><td>564</td></tr><tr><td>10</td><td>Wheat</td><td>50</td><td>162</td></tr><tr><td>11</td><td>Woods</td><td>50</td><td>1244</td></tr><tr><td>12</td><td>Buildings Grass Trees Drives</td><td>50</td><td>330</td></tr><tr><td>13</td><td>Stone Steel Towers</td><td>50</td><td>45</td></tr><tr><td>14</td><td>Alfalfa</td><td>15</td><td>39</td></tr><tr><td>15</td><td>Grass Pasture Mowed</td><td>15</td><td>11</td></tr><tr><td>16</td><td>Oats</td><td>15</td><td>5</td></tr></tbody><tfoot><tr><td colspan="2">Total</td><td>695</td><td>9671</td></tr></tfoot></table>

<table><caption>TABLE III
LAND-COVER CLASSES OF THE PAVIA UNIVERSITY DATASET, WITH THE STANDARD TRAINING AND TESTING SETS FOR EACH CLASS.</caption><thead><tr><th>Class No.</th><th>Class Name</th><th>Training</th><th>Testing</th></tr></thead><tbody><tr><td>1</td><td>Asphalt</td><td>548</td><td>6304</td></tr><tr><td>2</td><td>Meadows</td><td>540</td><td>18146</td></tr><tr><td>3</td><td>Gravel</td><td>392</td><td>1815</td></tr><tr><td>4</td><td>Trees</td><td>524</td><td>2912</td></tr><tr><td>5</td><td>Metal Sheets</td><td>265</td><td>1113</td></tr><tr><td>6</td><td>Bare Soil</td><td>532</td><td>4572</td></tr><tr><td>7</td><td>Bitumen</td><td>375</td><td>981</td></tr><tr><td>8</td><td>Bricks</td><td>514</td><td>3364</td></tr><tr><td>9</td><td>Shadows</td><td>231</td><td>795</td></tr></tbody><tfoot><tr><td colspan="2">Total</td><td>3921</td><td>40002</td></tr></tfoot></table>

3) **Houston2013 Data:** The third dataset was acquired by the ITRES CASI-1500 sensor over the campus of the University of Houston and its neighboring rural areas, Texas, USA, which, as a benchmark dataset, has been widely used for evaluating the performance of land cover classification [37]. The HS cube comprises of 349 × 1905 pixels with 144 wavelength bands in the range of 364nm to 1046nm at 10nm intervals. It is worth noting, however, that the Houston2013 dataset we used is a cloud-free version, which is processed to recover the missing data or remove occlusions by generating illumination-related threshold maps<fnref n="1" />. Table IV lists 15 challenging land-cover and land-use categories as well as the corresponding sample number of training and testing sets. Similarly, the visualization results with respect to the false-color HS image and the released training and testing samples provided by the 2013 IEEE GRSS data fusion contest<fnref n="2" /> are given in Fig. 10.

<footnote n="1">¹The data were provided by Prof. N. Yokoya from the University of Tokyo and RIKEN AIP.</footnote>

<footnote n="2">²http://www.grss.ieee.org/community/technical-committees/data-fusion/2013-ieee-grss-data-fusion-contest/</footnote>
<table><caption>TABLE IV
LAND-COVER CLASSES OF THE HOUSTON2013 DATASET, WITH THE STANDARD TRAINING AND TESTING SETS FOR EACH CLASS.</caption><thead><tr><th>Class No.</th><th>Class Name</th><th>Training</th><th>Testing</th></tr></thead><tbody><tr><td>1</td><td>Healthy Grass</td><td>198</td><td>1053</td></tr><tr><td>2</td><td>Stressed Grass</td><td>190</td><td>1064</td></tr><tr><td>3</td><td>Synthetic Grass</td><td>192</td><td>505</td></tr><tr><td>4</td><td>Tree</td><td>188</td><td>1056</td></tr><tr><td>5</td><td>Soil</td><td>186</td><td>1056</td></tr><tr><td>6</td><td>Water</td><td>182</td><td>143</td></tr><tr><td>7</td><td>Residential</td><td>196</td><td>1072</td></tr><tr><td>8</td><td>Commercial</td><td>191</td><td>1053</td></tr><tr><td>9</td><td>Road</td><td>193</td><td>1059</td></tr><tr><td>10</td><td>Highway</td><td>191</td><td>1036</td></tr><tr><td>11</td><td>Railway</td><td>181</td><td>1054</td></tr><tr><td>12</td><td>Parking Lot1</td><td>192</td><td>1041</td></tr><tr><td>13</td><td>Parking Lot2</td><td>184</td><td>285</td></tr><tr><td>14</td><td>Tennis Court</td><td>181</td><td>247</td></tr><tr><td>15</td><td>Running Track</td><td>187</td><td>473</td></tr><tr><td></td><td>Total</td><td>2832</td><td>12197</td></tr></tbody></table>

## B. Experimental Setup

1) **Evaluation Metrics:** We evaluate the classification performance of each model quantitatively in terms of three commonly-used indices, i.e., Overall Accuracy (OA), Average Accuracy (AA), Kappa Coefficient (κ). Moreover, the classification maps obtained by different models are visualized to make a qualitative comparison.

2) **Comparison with State-of-the-art Backbone Networks:** Several representative baselines and backbone networks are selected for the following comparison experiments. They are K-nearest neighbor (KNN), support vector machine (SVM), random forest (RF), 1-D CNN [17], 2-D CNN [22], RNN [23], miniGCN [26], transformers [27], and the proposed SpectralFormer. The parameter configurations of these compared methods are detailed below:

* For the KNN, the number of nearest neighbors (K) is an important factor, which greatly impacts the classification performance. We set it to 10.

* For the RF, 200 decision trees are used in our experiments.

* For the SVM, the libsvm toolbox<fnref n="3" /> is selected for the implementation of the HS image classification task. The SVM is performed by using the radial basis function (RBF) kernel. In RBF, two hyperparameters σ and λ can be optimally determined by five-fold cross validation on the training set in the range of σ = [2⁻<fnref n="3" />, 2⁻², ..., 2⁴] and λ = [10⁻², 10⁻¹, ..., 10⁴], respectively.

* For the 1-D CNN, one convolutional block is defined as the basic network unit, including a set of 1-D convolutional filters with the output size of 128, a batch normalization (BN) layer, and a ReLU activation function. A softmax function is finally added on the top layer of the 1-D CNN.

* The 2-D CNN architecture has three 2-D convolutional blocks and a softmax layer. Similar to the 1-D CNN, each convolutional block of 2-D CNN consists of a 2-D conventional layer, a BN layer, a max-pooling layer, and

a ReLU activation function. Moreover, the spatially and spectrally receptive fields in each 2-D convolutional layer are 3 × 3 × 32, 3 × 3 × 64, and 1 × 1 × 128, respectively.

* For the RNN, there are two recurrent layers with the gated recurrent unit (GRU). Each of them has 128 neuron units. The used codes are openly available from https://github.com/danfenghong/HyFTech.

* For the miniGCN, the network block successively contains a BN layer, a graph convolutional layer with 128 neuron units, and a ReLU layer. Note that the adjacency matrix in GCN can be generated using a KNN-based graph (the number of K is the same with the KNN classifier, i.e., K = 10). The miniGCN and 1-D CNN share the same network architecture (for a fair comparison). As the name suggests, the miniGCN can be trained in a mini-batch fashion. We refer to [26] for more details and the codes<fnref n="4" /> for the sake of reproducibility.

* For the transformers<fnref n="5" />, we follow the ViT network architecture [33], i.e., only including transformer encoders. In detail, five encoder blocks are used in the ViT-based network for HS image classification.

* For the proposed SpectralFormer, we adopt the same backbone architecture as the above transformers for a fair comparison. More specifically, the embedded spectrum with 64 units are fed into 5 cascaded transformer encoder blocks for HS image classification. Each encoder block consists of a four-head SA layer, a MLP with 8 hidden dimensions, and a GELU [38] nonlinear activation layer. The dropout layer is employed after encoding positional embeddings and in MLPs for inhibiting 10% neurons. Considering the fact that the parameter size is evidently increased from the pixel-wise SpectralFormer to the patch-wise one (the patch size is empirically set as 7 × 7), we additionally employ an ℓ₂ weight decay regularization [39] parameterized by 5e⁻<fnref n="3" /> to prevent a potential overfitting risk for the latter.

3) **Implementation Details:** Our proposed SpectralFormer was implemented on the PyTorch platform using a workstation with i7-6850K CPU, 128GB RAM, and an NVIDIA GTX 1080Ti 11GB GPU. We adopt the Adam optimizer [40] with a mini-batch size of 64. The learning rate is initialized with 5e⁻<fnref n="4" /> and decayed by multiplying a factor of 0.9 after each one-tenth of total epochs. We roughly set the epochs on the three datasets to 1000⁶. It is worth noting, however, that we found in practice our SpectralFormer with the CAF module is capable of embracing an evident efficiency improvement by convergence using much less epochs, i.e., 300 epochs for the Indian Pines dataset, and 600 epochs for the other two datasets.

4) **Computational Complexity Analysis:** For a given HS image with spectral length m, the per-layer computational complexity of the proposed SpectralFormer is mainly dominated by self-attention and multi-head operations that require

<footnote n="4">⁴https://github.com/danfenghong/IEEE_TGRS_GCN</footnote>

<footnote n="5">⁵The codes of transformers will be provided by authors, making it applicable to the HS image classification task.</footnote>

<footnote n="6">⁶The epochs might be slightly adjusted for different algorithms and different datasets, which will be specifically provided in our codes.</footnote>

<footnote n="3">³https://www.csie.ntu.edu.tw/~cjlin/libsvm/</footnote>
<table><caption>TABLE V
ABLATION ANALYSIS OF THE PROPOSED SPECTRALFORMER WITH A COMBINATION OF DIFFERENT MODULES ON THE INDIAN PINES DATASET.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Input</th><th colspan="2">Module</th><th colspan="3">Metric</th></tr><tr><th>GSE</th><th>CAF</th><th>OA (%)</th><th>AA (%)</th><th>κ</th></tr></thead><tbody><tr><td>Transformers (ViT)</td><td>pixel-wise</td><td>✓</td><td>✓</td><td>71.86</td><td>78.97</td><td>0.6804</td></tr><tr><td>SpectralFormer</td><td>pixel-wise</td><td>✓</td><td>✓</td><td>75.05</td><td>82.91</td><td>0.7175</td></tr><tr><td>SpectralFormer</td><td>pixel-wise</td><td>✓</td><td>✓</td><td>74.37</td><td>80.87</td><td>0.7084</td></tr><tr><td>SpectralFormer</td><td>pixel-wise</td><td>✓</td><td>✓</td><td>78.55</td><td>84.68</td><td>0.7554</td></tr><tr><td>SpectralFormer</td><td>patch-wise</td><td>✓</td><td>✓</td><td><strong>81.76</strong></td><td><strong>87.81</strong></td><td><strong>0.7919</strong></td></tr></tbody></table>

<table><caption>TABLE VI
SENSITIVITY ANALYSIS OF THE NUMBER OF NEIGHBORING BANDS IN SPECTRALFORMER (ONLY GSE AND GSE + CAF) IN TERMS OF OA, AA, AND κ ON THE INDIAN PINES DATASET.</caption><thead><tr><th rowspan="2">SpectralFormer</th><th rowspan="2">Metric</th><th colspan="6">The number of neighboring bands</th></tr><tr><th>1</th><th>3</th><th>5</th><th>7</th><th>9</th><th>11</th></tr></thead><tbody><tr><td rowspan="3">Only GSE</td><td>OA (%)</td><td>71.86</td><td>74.21</td><td>72.77</td><td><strong>75.05</strong></td><td>74.13</td><td>74.11</td></tr><tr><td>AA (%)</td><td>78.97</td><td>83.68</td><td>82.02</td><td><strong>82.91</strong></td><td>83.54</td><td>82.48</td></tr><tr><td>κ</td><td>0.6804</td><td>0.7088</td><td>0.6918</td><td><strong>0.7175</strong></td><td>0.7074</td><td>0.7067</td></tr><tr><td rowspan="3">GSE + CAF</td><td>OA (%)</td><td>74.37</td><td><strong>78.55</strong></td><td>77.11</td><td>77.41</td><td>76.46</td><td>77.65</td></tr><tr><td>AA (%)</td><td>80.87</td><td><strong>84.68</strong></td><td>84.55</td><td>84.20</td><td>83.45</td><td>83.85</td></tr><tr><td>κ</td><td>0.7084</td><td><strong>0.7554</strong></td><td>0.7394</td><td>0.7426</td><td>0.7319</td><td>0.7457</td></tr></tbody></table>

<table><caption>TABLE VII
CLASSIFICATION PERFORMANCE ANALYSIS BETWEEN DIFFERENT SCs IN TRANSFORMERS ON THE INDIAN PINES DATASET.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Input</th><th rowspan="2">SC</th><th colspan="3">Metric</th></tr><tr><th>OA (%)</th><th>AA (%)</th><th>κ</th></tr></thead><tbody><tr><td>Transformers (ViT)</td><td rowspan="3">pixel-wise</td><td>short-range</td><td>71.86</td><td>78.97</td><td>0.6804</td></tr><tr><td>Transformers (ViT)</td><td>long-range</td><td>67.70</td><td>73.12</td><td>0.6315</td></tr><tr><td>Transformers (ViT)</td><td>CAF</td><td><strong>74.37</strong></td><td><strong>80.87</strong></td><td><strong>0.7084</strong></td></tr></tbody></table>

an overall $O(m^2d + md^2)$, where $d$ is the size of the hidden features that are also used in deep competitors for a fair theoretical comparison. The RNN yields the complexity of $O(md^2)$ after $m$ times sequential operations, while a CNN with the kernel width $k$ increases it considerably to $O(kmd^2)$. The GCN (i.e., miniGCN) requires $O(bmd + b^2m)$ owing to its batch-wise graph sampling, where $b$ denotes the size of mini-batches.

### C. Model Analysis

1) **Ablation Study:** We investigate the performance gain of the proposed SpectralFormer in terms of classification accuracies by adding (in stepwise fashion) different modules (i.e., GSE and CAF) in networks. For that, we conduct extensive ablation experiments on the Indian Pines dataset to verify the effectiveness of these components (or modules) in SpectralFormer for HS image classification applications, as listed in Table V.

In detail, the classic transformers (ViT) without GSE and CAF modules yield the lowest classification accuracies, which to some extent indicates that the ViT architecture might not be a good fit for the HS image classification. By plugging either GSE or CAF into ViT, the classification results of pixel-wise SpectralFormer are better than that of ViT (beyond around 4% and 3% OAs, respectively). What is better still, the joint exploitation of GSE and CAF can further bring a dramatic performance improvement (more than 4% OA). More remarkably, our SpectralFormer is also capable of capturing the locally spatial semantics of HS images by simply unfolding the

patch-wise input. As a result, the patch-wise SpectralFormer performs observably better than pixel-wise ones at an increase at least 3% OA (compared to the second record, i.e., 78.55%).

Interestingly, there is a noteworthy trend in Table V. That is, the joint use of GSE and CAF tends to obtain the best performance when only using a smaller number of neighboring bands in GSE, compared to that of only using GSE. This might be well explained by that after adding CAF, the spectral information is capable of being learned more efficiently and easier. In other words, the less overlap between spectral bands (i.e., the number of neighboring bands is smaller in GSE) could be enough to obtain a better classification performance, after the CAF module is activated.

2) **Parameter Sensitivity Analysis:** Apart from the learnable parameters in networks and hyper-parameters required in the training process, the number of neighboring bands in GES plays a vital role in the final classification performance. It is, therefore, indispensable to explore the proper parameter range. Similarly, we investigate the parameter sensitivity on the Indian pines dataset in an ablation manner, i.e., only with GSE and the joint use of GSE and CAF. Table VI lists the changing trend of the classification accuracies with the gradual increase of the number of grouped bands in terms of OA, AA, and $κ$. A common conclusion is that GSE can better excavate subtle spectral discrepancies by effectively capturing locally spectral embeddings from neighboring bands. Within a certain range, the parameter is not sensitive to the classification performance. This provides great potentials of the proposed model in the practical applications. In other words, the parameter can be simply and directly used for other datasets.

We also make a quantitative comparison among short-range SC (e.g., in ResNet), long-range SC (e.g., in U-Net), and middle-range (i.e., our CAF module) SC in transformers, in order to verify the effectiveness of the proposed SpectralFormer in processing spectral data. Table VII quantifies the classification performance comparison of using short-, middle-, and long-range SCs, respectively, on the Indian Pines dataset. In general, the ViT with long-range SC yields a poor performance, possibly since such an SC strategy might fail to sufficiently fuse and convey long-range cross-layer features, tending to lose partially “important” information. This demonstrates the superiority of the CAF module that can exchange information across different layers more effectively (cf. short-range SC) and reduce the information loss (cf. long-range SC).

Furthermore, we randomly selected varying number of training samples from the given training set on the Indian Pines dataset out of 10 runs in the proportion of [10%, 20%, ..., 100%] at intervals of 10%. The average results with standard deviation values in terms of the OA obtained by the proposed SpectralFormer (including pixel-wise and patch-wise versions) are reported in Fig. 6. There is a basically reasonable trend in OA's results (see Fig. 6). That is, with the increase of the percentage of used training samples, the classification performance gradually improves. Note that the OAs are tending towards stability when more training samples (e.g., 80%, 90%, and 100%) are involved, showing the stability of the proposed SpectralFormer to a great extent. Moreover,
<table><caption>TABLE VIII
QUANTITATIVE PERFORMANCE OF DIFFERENT CLASSIFICATION METHODS IN TERMS OF OA, AA, AND $\kappa$ AS WELL AS THE ACCURACIES FOR EACH CLASS ON THE INDIAN PINES DATASET. THE BEST ONE IS SHOWN IN BOLD.</caption><thead><tr><th rowspan="2">Class No.</th><th colspan="3">Conventional Classifiers</th><th colspan="4">Classic Backbone Networks</th><th rowspan="2">Transformers (ViT)</th><th colspan="2">SpectralFormer</th></tr><tr><th>KNN</th><th>RF</th><th>SVM</th><th>1-D CNN</th><th>2-D CNN</th><th>RNN</th><th>miniGCN</th><th>pixel-wise</th><th>patch-wise</th></tr></thead><tbody><tr><td>1</td><td>45.45</td><td>57.80</td><td>67.34</td><td>47.83</td><td>65.90</td><td>69.00</td><td><strong>72.54</strong></td><td>53.25</td><td>62.64</td><td>70.52</td></tr><tr><td>2</td><td>46.94</td><td>56.51</td><td>67.86</td><td>42.35</td><td>76.66</td><td>58.93</td><td>55.99</td><td>66.20</td><td>66.20</td><td><strong>81.89</strong></td></tr><tr><td>3</td><td>77.72</td><td>80.98</td><td><strong>93.48</strong></td><td>60.87</td><td>92.39</td><td>77.17</td><td>92.93</td><td>86.41</td><td>88.59</td><td>91.30</td></tr><tr><td>4</td><td>84.56</td><td>85.68</td><td>94.63</td><td>89.49</td><td>93.96</td><td>82.33</td><td>92.62</td><td>89.71</td><td>90.16</td><td><strong>95.53</strong></td></tr><tr><td>5</td><td>80.06</td><td>79.34</td><td>88.52</td><td>92.40</td><td>87.23</td><td>67.72</td><td><strong>94.98</strong></td><td>87.66</td><td>89.24</td><td>85.51</td></tr><tr><td>6</td><td>97.49</td><td>95.44</td><td>94.76</td><td>97.04</td><td>97.27</td><td>89.07</td><td>98.63</td><td>89.98</td><td>95.90</td><td><strong>99.32</strong></td></tr><tr><td>7</td><td>64.81</td><td>77.56</td><td>73.86</td><td>59.69</td><td>77.23</td><td>69.06</td><td>64.71</td><td>72.22</td><td><strong>85.19</strong></td><td>81.81</td></tr><tr><td>8</td><td>48.68</td><td>58.85</td><td>52.07</td><td>65.38</td><td>57.03</td><td>63.56</td><td>68.78</td><td>66.00</td><td>74.48</td><td><strong>75.48</strong></td></tr><tr><td>9</td><td>44.33</td><td>62.23</td><td>72.70</td><td><strong>93.44</strong></td><td>72.87</td><td>65.07</td><td>69.33</td><td>57.09</td><td>72.34</td><td>73.76</td></tr><tr><td>10</td><td>96.30</td><td>95.06</td><td>98.77</td><td>99.38</td><td><strong>100.00</strong></td><td>95.06</td><td>98.77</td><td>97.53</td><td>98.15</td><td>98.77</td></tr><tr><td>11</td><td>74.28</td><td>88.75</td><td>86.17</td><td>84.00</td><td>92.85</td><td>88.67</td><td>87.78</td><td>87.62</td><td>93.01</td><td><strong>93.17</strong></td></tr><tr><td>12</td><td>15.45</td><td>54.24</td><td>71.82</td><td>86.06</td><td><strong>88.18</strong></td><td>50.00</td><td>50.00</td><td>63.94</td><td>60.91</td><td>78.48</td></tr><tr><td>13</td><td>91.11</td><td>97.78</td><td>95.56</td><td>91.11</td><td><strong>100.00</strong></td><td>97.78</td><td><strong>100.00</strong></td><td>95.56</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td></tr><tr><td>14</td><td>33.33</td><td>56.41</td><td>82.05</td><td>84.62</td><td>84.62</td><td>66.67</td><td>48.72</td><td>79.49</td><td><strong>87.18</strong></td><td>79.49</td></tr><tr><td>15</td><td>81.82</td><td>81.82</td><td>90.91</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>81.82</td><td>72.73</td><td>90.91</td><td>90.91</td><td><strong>100.00</strong></td></tr><tr><td>16</td><td>40.00</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>80.00</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>80.00</td><td>80.00</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td></tr><tr><td>OA (%)</td><td>59.17</td><td>69.80</td><td>72.36</td><td>70.43</td><td>75.89</td><td>70.66</td><td>75.11</td><td>71.86</td><td>78.55</td><td><strong>81.76</strong></td></tr><tr><td>AA (%)</td><td>63.90</td><td>76.78</td><td>83.16</td><td>79.60</td><td>86.64</td><td>76.37</td><td>78.03</td><td>78.97</td><td>84.68</td><td><strong>87.81</strong></td></tr><tr><td>$\kappa$</td><td>0.5395</td><td>0.6591</td><td>0.6888</td><td>0.6642</td><td>0.7281</td><td>0.6673</td><td>0.7164</td><td>0.6804</td><td>0.7554</td><td><strong>0.7919</strong></td></tr></tbody></table>

<table><caption>TABLE IX
QUANTITATIVE PERFORMANCE OF DIFFERENT CLASSIFICATION METHODS IN TERMS OF OA, AA, AND $\kappa$, AS WELL AS THE ACCURACIES FOR EACH CLASS ON THE PAVIA UNIVERSITY DATASET. THE BEST ONE IS SHOWN IN BOLD.</caption><thead><tr><th rowspan="2">Class No.</th><th colspan="3">Conventional Classifiers</th><th colspan="4">Classic Backbone Networks</th><th rowspan="2">Transformers (ViT)</th><th colspan="2">SpectralFormer</th></tr><tr><th>KNN</th><th>RF</th><th>SVM</th><th>1-D CNN</th><th>2-D CNN</th><th>RNN</th><th>miniGCN</th><th>pixel-wise</th><th>patch-wise</th></tr></thead><tbody><tr><td>1</td><td>73.86</td><td>79.81</td><td>74.22</td><td>88.90</td><td>80.98</td><td>84.01</td><td><strong>96.35</strong></td><td>71.51</td><td>82.95</td><td>82.73</td></tr><tr><td>2</td><td>64.31</td><td>54.90</td><td>52.79</td><td>58.81</td><td>81.70</td><td>66.95</td><td>89.43</td><td>76.82</td><td><strong>95.23</strong></td><td>94.03</td></tr><tr><td>3</td><td>55.10</td><td>46.34</td><td>65.45</td><td>73.11</td><td>67.99</td><td>58.46</td><td><strong>87.01</strong></td><td>46.39</td><td>78.18</td><td>73.66</td></tr><tr><td>4</td><td>94.95</td><td><strong>98.73</strong></td><td>97.42</td><td>82.07</td><td>97.36</td><td>97.70</td><td>94.26</td><td>96.39</td><td>87.95</td><td>93.75</td></tr><tr><td>5</td><td>99.19</td><td>99.01</td><td>99.46</td><td>99.46</td><td>99.64</td><td>99.10</td><td><strong>99.82</strong></td><td>99.19</td><td>99.46</td><td>99.28</td></tr><tr><td>6</td><td>65.16</td><td>75.94</td><td>93.48</td><td><strong>97.92</strong></td><td>97.59</td><td>83.18</td><td>43.12</td><td>83.18</td><td>65.84</td><td>90.75</td></tr><tr><td>7</td><td>84.30</td><td>78.70</td><td>87.87</td><td>88.07</td><td>82.47</td><td>83.08</td><td><strong>90.96</strong></td><td>83.08</td><td>92.35</td><td>87.56</td></tr><tr><td>8</td><td>84.10</td><td>90.22</td><td>89.39</td><td>88.14</td><td><strong>97.62</strong></td><td>89.63</td><td>77.42</td><td>89.63</td><td>85.26</td><td>95.81</td></tr><tr><td>9</td><td>98.36</td><td>97.99</td><td>99.87</td><td>99.87</td><td>95.60</td><td>96.48</td><td>87.27</td><td>96.48</td><td><strong>100.00</strong></td><td>94.21</td></tr><tr><td>OA (%)</td><td>70.53</td><td>69.67</td><td>70.82</td><td>75.50</td><td>86.05</td><td>77.13</td><td>79.79</td><td>76.99</td><td>87.94</td><td><strong>91.07</strong></td></tr><tr><td>AA (%)</td><td>79.68</td><td>80.18</td><td>84.44</td><td>86.26</td><td>88.99</td><td>84.29</td><td>85.07</td><td>80.22</td><td>87.47</td><td><strong>90.20</strong></td></tr><tr><td>$\kappa$</td><td>0.6268</td><td>0.6237</td><td>0.6423</td><td>0.6948</td><td>0.8187</td><td>0.7101</td><td>0.7367</td><td>0.7010</td><td>0.8358</td><td><strong>0.8805</strong></td></tr></tbody></table>

the patch-wise SpectralFormer observably outperforms the pixel-wise one, as expected.

## D. Quantitative Results and Analysis

Quantitative classification results in terms of three overall indices, i.e., OA, AA, and $\kappa$, and the accuracies for each class are reported in Tables VIII, IX, and X for Indian Pines, Pavia University, and Houston2013 HS datasets, respectively.

Overall, the conventional classifiers, e.g., KNN, RF, SVM, achieve similar classification performance on all three data sets, except the accuracies in terms of OA, AA, and $\kappa$ with KNN on the Indian Pines dataset (which are far inferior to those using RF and SVM). Owing to the powerful learning ability of DL techniques, classic backbone networks, e.g., 1-D CNN, 2-D CNN, RNN, miniGCN, observably perform better

than aforementioned conventional classifiers, i.e., KNN, RF, SVM. The results, to a great extent, demonstrate the value and practicality of DL-based approaches in HS image classification. Without any convolutional and recurrent operations, transformers extract finer spectral representations from the sequence perspectives, yielding a comparable performance to CNNs-, RNNs-, or GCNs-based models.

Although transformers are capable of capturing globally sequential information, the ability in modeling some key factors – locally spectral discrepancies – remains limited, leads to a performance bottleneck. To overcome this problem, the proposed SpectralFormer fully extracts local spectral information from neighboring bands, dramatically improving the classification performance. In particular, the pixel-wise SpectralFormer unexpectedly outperforms others, even
<table><caption>TABLE X
QUANTITATIVE PERFORMANCE OF DIFFERENT CLASSIFICATION METHODS IN TERMS OF OA, AA, AND κ AS WELL AS THE ACCURACIES FOR EACH CLASS ON THE HOUSTON2013 DATASET. THE BEST ONE IS SHOWN IN BOLD.</caption><thead><tr><th rowspan="2">Class No.</th><th colspan="3">Conventional Classifiers</th><th colspan="4">Classic Backbone Networks</th><th rowspan="2">Transformers (ViT)</th><th colspan="2">SpectralFormer</th></tr><tr><th>KNN</th><th>RF</th><th>SVM</th><th>1-D CNN</th><th>2-D CNN</th><th>RNN</th><th>miniGCN</th><th>pixel-wise</th><th>patch-wise</th></tr></thead><tbody><tr><td>1</td><td>83.19</td><td>83.38</td><td>83.00</td><td>87.27</td><td>85.09</td><td>82.34</td><td><strong>98.39</strong></td><td>82.81</td><td>83.48</td><td>81.86</td></tr><tr><td>2</td><td>95.68</td><td>98.40</td><td>98.40</td><td>98.21</td><td>99.91</td><td>94.27</td><td>92.11</td><td>96.62</td><td>95.58</td><td><strong>100.00</strong></td></tr><tr><td>3</td><td>99.41</td><td>98.02</td><td>99.60</td><td><strong>100.00</strong></td><td>77.23</td><td>99.60</td><td>99.60</td><td>99.80</td><td>99.60</td><td>95.25</td></tr><tr><td>4</td><td>97.92</td><td>97.54</td><td>98.48</td><td>92.99</td><td>97.73</td><td>97.54</td><td>96.78</td><td><strong>99.24</strong></td><td>99.15</td><td>96.12</td></tr><tr><td>5</td><td>96.12</td><td>96.40</td><td>97.82</td><td>97.35</td><td><strong>99.53</strong></td><td>93.28</td><td>97.73</td><td>97.73</td><td>97.44</td><td><strong>99.53</strong></td></tr><tr><td>6</td><td>92.31</td><td><strong>97.20</strong></td><td>90.91</td><td>95.10</td><td>92.31</td><td>95.10</td><td>95.10</td><td>95.10</td><td>95.10</td><td>94.41</td></tr><tr><td>7</td><td>80.88</td><td>82.09</td><td>90.39</td><td>77.33</td><td><strong>92.16</strong></td><td>83.77</td><td>57.28</td><td>76.77</td><td>88.99</td><td>83.12</td></tr><tr><td>8</td><td>48.62</td><td>40.65</td><td>40.46</td><td>51.38</td><td><strong>79.39</strong></td><td>56.03</td><td>68.09</td><td>55.65</td><td>73.31</td><td>76.73</td></tr><tr><td>9</td><td>72.05</td><td>69.78</td><td>41.93</td><td>27.95</td><td><strong>86.31</strong></td><td>72.14</td><td>53.92</td><td>67.42</td><td>71.86</td><td>79.32</td></tr><tr><td>10</td><td>53.19</td><td>57.63</td><td>62.64</td><td><strong>90.83</strong></td><td>43.73</td><td>84.17</td><td>77.41</td><td>68.05</td><td>87.93</td><td>78.86</td></tr><tr><td>11</td><td>86.24</td><td>76.09</td><td>75.43</td><td>79.32</td><td>87.00</td><td>82.83</td><td>84.91</td><td>82.35</td><td>80.36</td><td><strong>88.71</strong></td></tr><tr><td>12</td><td>44.48</td><td>49.38</td><td>60.04</td><td>76.56</td><td>66.28</td><td>70.61</td><td>77.23</td><td>58.50</td><td>70.70</td><td><strong>87.32</strong></td></tr><tr><td>13</td><td>28.42</td><td>61.40</td><td>49.47</td><td>69.47</td><td><strong>90.18</strong></td><td>69.12</td><td>50.88</td><td>60.00</td><td>71.23</td><td>72.63</td></tr><tr><td>14</td><td>97.57</td><td><strong>99.60</strong></td><td>98.79</td><td>99.19</td><td>90.69</td><td>98.79</td><td>98.38</td><td>98.79</td><td>98.79</td><td><strong>100.00</strong></td></tr><tr><td>15</td><td>98.10</td><td>97.67</td><td>97.46</td><td>98.10</td><td>77.80</td><td>95.98</td><td>98.52</td><td>98.73</td><td>98.73</td><td><strong>99.79</strong></td></tr><tr><td>OA (%)</td><td>77.30</td><td>77.48</td><td>76.91</td><td>80.04</td><td>83.72</td><td>83.23</td><td>81.71</td><td>80.41</td><td>86.14</td><td><strong>88.01</strong></td></tr><tr><td>AA (%)</td><td>78.28</td><td>80.35</td><td>78.99</td><td>82.74</td><td>84.35</td><td>85.04</td><td>83.09</td><td>82.50</td><td>87.48</td><td><strong>88.91</strong></td></tr><tr><td>κ</td><td>0.7538</td><td>0.7564</td><td>74.94</td><td>0.7835</td><td>0.8231</td><td>0.8183</td><td>0.8018</td><td>0.7876</td><td>0.8497</td><td><strong>0.8699</strong></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Fig. 6. Classification results (OA) obtained by our proposed SpectralFormer (pixel-wise and patch-wise) with the varying number of used training samples on the Indian Pines dataset.</figcaption></figure>

though comparing to CNNs-based methods considering spatial contents. Undoubtedly, the patch-wise SpectralFormer obtains higher classification accuracies which are far superior to those obtained by other competitors, since the spatial-contextual information is jointly considered in the sequential feature extraction process.

In addition, for those challenging classes that have limited training samples (e.g., *Grass Pasture Mowed, Oats*) and imbalanced (or noisy) samples (e.g., *Corn Mintill, Grass Pasture, Hay Windrowed*) on the Indian Pines data, SpectralFormer, either pixel-wise or patch-wise input, tends to obtain better classification performance by focusing on particular absorption positions of spectral profiles. By contrary, despite the excellent representation ability for spectral sequence data, transformers

(ViT) fail to accurately capture the detailed spectral absorption or change due to the weak modeling ability in locally spectral discrepancies.

## E. Visual Evaluation

We make a qualitative evaluation by visualizing the classification maps obtained by different methods. Figs. 7, 8, and 10 provide the obtained results for the Indian Pines, Pavia University, and Houston2013 datasets, respectively. Roughly, conventional classification models (e.g., KNN, RF, SVM) tend to generate salt and pepper noises in classification maps of three considered datasets. This indirectly indicates that these classifiers fail to accurately identifying the materials of objects. Not surprisingly, DL-based models, e.g., CNNs, RNNs, GCNs, obtain relatively smooth classification maps, owing to their powerful nonlinear data fitting ability. As an emerging network architecture, transformers (ViT used in our case) can extract highly sequential representations from HS images, leading to visualized classification maps that are comparable to the above classic backbone networks. By enhancing spectrally neighboring information and conveying “memory” information across layers more effectively, our proposed SpectralFormer obtains highly desirable classification maps, especially in terms of texture and edge details. Furthermore, we selected a region of interest (ROI) (from Figs. 7, 8, and 10) zoomed-in 2 times to highlight the differences in classification maps between different models, further evaluating their classification performance more intuitively. As can be seen from these ROIs, a remarkable phenomenon is that our methods, i.e., pixel-wise and patch-wise SpectralFormers, show more realistic and finer details. In particular, the results of our methods have less noisy points compared to those pixel-wise methods, e.g., KNN, RF, SVM, 1-D CNN, RNN, miniGCN, and ViT, but also avoid over-smoothness in edges or some small semantic objects
<figure><img src="image_8.png" /><figcaption>Fig. 7. Spatial distribution of training and testing sets, and the classification maps obtained by different models on the Indian Pines dataset, where a ROI zoomed in 2 times is highlighted for more detailed observation.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8. Spatial distribution of training and testing sets, and the classification maps obtained by different models on the Pavia University dataset, where a ROI zoomed in 2 times is highlighted for more detailed observation.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9. Visualization of selected encoder output features obtained with only CAF and without CAF (i.e., the original ViT).</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. 10. Spatial distribution of training and testing sets, and the classification maps obtained by different models on the Houston2013 dataset, where a ROI zoomed in 2 times is highlighted for more detailed observation.</figcaption></figure>

(cf. 2-D CNN), which yields more accurate classification performance.

**Feature Visualization.** Fig. 9 visualizes selected encoder output features using the proposed SpectralFormer framework with only CAF and without CAF (i.e., ViT). We selectively pick out some representative feature maps for visual comparison, where the visualization results of using the CAF module have finer appearance (e.g., the edge or outline of objects, textural structure, etc.) than those without CAF. This also demonstrates the effectiveness and superiority of the designed CAF module from the visual perspective.

## IV. CONCLUSION

HS images are typically collected (or represented) as a data cube with spatial-spectral information, which can be generally regarded as a sequence of data along the spectral dimension. Unlike CNNs, that focus mainly on contextual information modeling, transformers have been proven to be a powerful

architecture in characterizing the sequential properties globally. However, the classic transformers-based vision networks, e.g., ViT, inevitably suffer from performance degradation when processing HS-like data. This might be explained well by the fact that ViT fails to model locally detailed spectral discrepancies and convey “memory”-like components (from shallow to deep layers) effectively. To this end, in this paper we propose a new transformers-based backbone network, called SpectralFormer, which is more focused on extracting spectral information. Without using any convolution or recurrent units, the proposed SpectralFormer can achieve state-of-the-art classification results for HS images.

In the future, we will investigate strategies to further improve the transformers-based architecture by utilizing more advanced techniques, e.g., attention, self-supervised learning, making it more applicable to the HS image classification task, and also attempt to establish a lightweight transformers-based network to reduce the network complexity while maintaining
its performance. Moreover, we would also like to embed more
physical characteristics of spectral bands and prior knowledge
of HS images into the proposed framework, yielding more
interpretable deep models. Furthermore, the number of skipped
and connected encoders in the CAF module is an important
factor that might be capable of improving the classification
performance of the proposed SpectralFormer, which should
be paid more attention to in future work.

REFERENCES

[1] D. Hong, W. He, N. Yokoya, J. Yao, L. Gao, L. Zhang, J. Chanussot, and X. X. Zhu, "Interpretable hyperspectral artificial intelligence: When non-convex modeling meets hyperspectral remote sensing," *IEEE Geosci. Remote Sens. Mag.*, vol. 9, no. 2, pp. 52–87, 2021.

[2] Y. Wang, J. Peng, Q. Zhao, Y. Leung, X.-L. Zhao, and D. Meng, "Hyperspectral image restoration via total variation regularized low-rank tensor decomposition," *IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.*, vol. 11, no. 4, pp. 1227–1243, 2017.

[3] W. Cao, K. Wang, G. Han, J. Yao, and A. Cichocki, “A robust pca approach with noise structure learning and spatial–spectral low-rank modeling for hyperspectral image restoration,” *IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.*, vol. 11, no. 10, pp. 3863–3879, 2018.

[4] M. Wang, Q. Wang, J. Chanussot, and D. Hong, “$l_0-l_1$ hybrid total variation regularization and its applications on hyperspectral image mixed noise removal and compressed sensing,” *IEEE Trans. Geosci. Remote Sens.*, 2021. DOI: 10.1109/TGRS.2021.3055516.

[5] J. Peng, W. Sun, H.-C. Li, W. Li, X. Meng, C. Ge, and Q. Du, “Low-rank and sparse representation for hyperspectral image processing: A review,” *IEEE Geosci. Remote Sens. Mag.*, 2021.

[6] D. Hong, N. Yokoya, J. Chanussot, J. Xu, and X. X. Zhu, “Joint and progressive subspace analysis (jpsa) with spatial-spectral manifold alignment for semi-supervised hyperspectral dimensionality reduction,” *IEEE Trans. Cybern.*, vol. 51, no. 7, pp. 3602–3615, 2021.

[7] F. Luo, T. Guo, Z. Lin, J. Ren, and X. Zhou, “Semisupervised hypergraph discriminant learning for dimensionality reduction of hyperspectral image,” *IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.*, vol. 13, pp. 4242–4256, 2020.

[8] J. Yao, D. Meng, Q. Zhao, W. Cao, and Z. Xu, “Nonconvex-sparsity and nonlocal-smoothness-based blind hyperspectral unmixing,” *IEEE Trans. Image Process.*, vol. 28, no. 6, pp. 2991–3006, 2019.

[9] D. Hong, N. Yokoya, J. Chanussot, and X. Zhu, “An augmented linear mixing model to address spectral variability for hyperspectral unmixing,” *IEEE Trans. Image Process.*, vol. 28, no. 4, pp. 1923–1938, 2019.

[10] Y. Yuan, Z. Zhang, and Q. Wang, “Improved collaborative non-negative matrix factorization and total variation for hyperspectral unmixing,” *IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.*, vol. 13, pp. 998–1010, 2020.

[11] L. Gao, Z. Han, D. Hong, B. Zhang, and J. Chanussot, “Cycunet: Cycle-consistency unmixing network by learning cascaded autoencoders,” *IEEE Trans. Geosci. Remote Sens.*, 2021. DOI: 10.1109/TGRS.2021.3064958.

[12] D. Hong, L. Gao, J. Yao, N. Yokoya, J. Chanussot, U. Heiden, and B. Zhang, “Endmember-guided unmixing network (egu-net): A general deep learning framework for self-supervised hyperspectral unmixing,” *IEEE Trans. Neural Netw. Learn. Syst.*, May 2021. DOI: 10.1109/TNNLS.2021.3082289.

[13] D. Hong, N. Yokoya, J. Chanussot, J. Xu, and X. Zhu, “Learning to propagate labels on graphs: An iterative multitask regression framework for semi-supervised hyperspectral dimensionality reduction,” *ISPRS J. Photogramm. Remote Sens.*, vol. 158, pp. 35–49, 2019.

[14] J. Peng, W. Sun, and Q. Du, “Self-paced joint sparse representation for the classification of hyperspectral images,” *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 2, pp. 1183–1194, 2018.

[15] D. Hong, X. Wu, P. Ghamisi, J. Chanussot, N. Yokoya, and X. X. Zhu, “Invariant attribute profiles: A spatial-frequency joint feature extractor for hyperspectral image classification,” *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 6, pp. 3791–3808, 2020.

[16] Q. Li, B. Zheng, B. Tu, J. Wang, and C. Zhou, “Ensemble emd-based spectral-spatial feature extraction for hyperspectral image classification,” *IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.*, vol. 13, pp. 5134–5148, 2020.

[17] B. Rasti, D. Hong, R. Hang, P. Ghamisi, X. Kang, J. Chanussot, and J. Benediktsson, “Feature extraction for hyperspectral imagery: The evolution from shallow to deep: Overview and toolbox,” *IEEE Geosci. Remote Sens. Mag.*, vol. 8, no. 4, pp. 60–88, 2020.

[18] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.

[19] X. Zhao, R. Tao, W. Li, H.-C. Li, Q. Du, W. Liao, and W. Philips, “Joint classification of hyperspectral and lidar data using hierarchical random walk and deep cnn architecture,” *IEEE Trans. Geosci. Remote Sens.*, vol. 58, no. 10, pp. 7355–7370, 2020.

[20] M. Zhang, W. Li, Q. Du, L. Gao, and B. Zhang, “Feature extraction for classification of hyperspectral and lidar data using patch-to-patch cnn,” *IEEE Trans. Cybern.*, vol. 50, no. 1, pp. 100–111, 2018.

[21] Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, “Deep learning-based classification of hyperspectral data,” *IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.*, vol. 7, no. 6, pp. 2094–2107, 2014.

[22] Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” *IEEE Trans. Geosci. Remote Sens.*, vol. 54, no. 10, pp. 6232–6251, 2016.

[23] R. Hang, Q. Liu, D. Hong, and P. Ghamisi, “Cascaded recurrent neural networks for hyperspectral image classification,” *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 8, pp. 5384–5394, 2019.

[24] L. Zhu, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Generative adversarial networks for hyperspectral image classification,” *IEEE Trans. Geosci. Remote Sens.*, vol. 56, no. 9, pp. 5046–5063, 2018.

[25] M. E. Paoletti, J. M. Haut, R. Fernandez-Beltran, J. Plaza, A. Plaza, J. Li, and F. Pla, “Capsule networks for hyperspectral image classification,” *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 4, pp. 2145–2160, 2018.

[26] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph convolutional networks for hyperspectral image classification,” *IEEE Trans. Geosci. Remote Sens.*, vol. 59, no. 7, pp. 5966–5978, 2021.

[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” *arXiv preprint arXiv:1706.03762*, 2017.

[28] S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 9, pp. 6690–6709, 2019.

[29] H. Abdi and L. J. Williams, “Principal component analysis,” *Wiley Interdiscip. Rev. Comput. Stat.*, vol. 2, no. 4, pp. 433–459, 2010.

[30] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” *IEEE Trans. Neural Netw.*, vol. 5, no. 2, pp. 157–166, 1994.

[31] G. Ke, D. He, and T.-Y. Liu, “Rethinking the positional encoding in language pre-training,” *arXiv preprint arXiv:2006.15595*, 2020.

[32] Y. Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” *arXiv preprint arXiv:2103.03404*, 2021.

[33] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” *arXiv preprint arXiv:2010.11929*, 2020.

[34] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in *Proc. CVPR*, pp. 7794–7803, 2018.

[35] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in *Proc. CVPR*, pp. 770–778, 2016.

[36] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in *Proc. MICCAI*, pp. 234–241, Springer, 2015.

[37] D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, D. Qian, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” *IEEE Trans. Geosci. Remote Sens.*, vol. 59, no. 5, pp. 4340–4354, 2021.

[38] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” *arXiv preprint arXiv:1606.08415*, 2016.

[39] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” *arXiv preprint arXiv:1711.05101*, 2017.

[40] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” *arXiv preprint arXiv:1412.6980*, 2014.