# Nested Transformers for Hyperspectral Image Classification

Authors: Zitong Zhang; Qiaoyu Ma; Heng Zhou; Na Gong
Journal: Journal of Sensors
 (2022)

---

## Abstract

Substantial deep learning methods have been utilized for hyperspectral image (HSI) classification recently. Vision Transformer (ViT) is skilled in modeling the overall structure of images and has been introduced to HSI classification task. However, the fixed patch division operation in ViT may lead to insufficient feature extraction, especially the features of the edges between patches will be ignored. To address this problem, we devise a workflow for HSI classification based on the Nested Transformers (NesT). The NesT employs the block aggregation module to extract edge information between patches, which realizes cross-block communication of nonlocal information and optimizes global information extraction. In this paper, the NesT is used for HSI classification for the first time. The experiments are carried out on four widely used hyperspectral datasets: Indian Pines, Salinas, Tea Farm, and Xiongan New Area (Matiwan Village). The obtained results reveal that the NesT can provide competitive results compared to conventional machine learning and deep learning methods and achieve top accuracy on four datasets, which proves the superiority of the NesT in HSI classification with limited training samples.


---

## Body

## 1. Introduction

Hyperspectral imaging technology generates multiple spatially aligned images of ground objects by the sensor while recording hundreds of contiguous narrow spectral bands in the spectral domain. Therefore, the hyperspectral image (HSI) is a data cube. Different from the conventional images, each pixel in the data cube is a 1D spectral representation of the response at a given spatial location, and each 2D layer is a spatial image representing the response at a specific wavelength band, so the information available from HSI is much richer and contains both spectral and spatial information [1–4].Most of the research explored for HSI classification can be divided into spectral classifiers and spectral-spatial classifiers. The spectral feature is the original feature of HSI, while the spatial feature refers to the relationship between the center pixel and its neighborhoods. In the early stages, researchers mainly focused on pure spectral feature-based methods, simply applying classifiers to pixels, such as support vector machine (SVM) [5] and logistic regression (LR) [6], while spectral classifiers cannot achieve good performance due to the existence of different objects with the same spectrum or different spectral features with the same object.Therefore, the spectral-spatial classifiers emerged to improve the accuracy by fusing spectral and spatial features. Spatial correlation analysis [7], Markov random fields [8], Gabor filter [9], and morphological features [10] are widely used in spatial feature extraction. However, the strong correlation between pixels results in information redundancy after feature extraction stage. It is a primary challenge to extract discriminative features from HSI that requires adequate approaches.Deep learning methods have become a powerful and popular tool in remote sensing community due to the excellent ability of feature extraction and generalization. Convolution-based model is one of the most commonly used deep learning methods for HSI classification, which obtains the image response to specific features at different positions by convolutional kernels as a classification basis [11–13]. Lee et al. [14] proposed a novel deep Convolutional Neural Network (CNN) to optimally explore local contextual interactions by jointly exploiting local spectral-spatial relationships of neighboring pixels. Feng et al. [15] presented a novel CNN method that considered multilayer spectral-spatial feature fusion and sample augmentation with local and nonlocal constraints. In [16], a CNN-based spatial feature fusion algorithm was employed to integrate spatial information with spectral features. In [17], a multidimensional hybrid convolutional architecture was used for extracting spectral-spatial features in two steps, which optimized the description of the spatial details. It can be learned from the above literature that convolution-based models have been used to learn feature representations from raw data and form more complex and abstract concepts successfully by combining simpler concepts, which was a hot spot in previous research.However, the convolutional structure extracts the features from HSI via the local receptive field, resulting in the lack of a description of the global features in HSI. The emerging Vision Transformer (ViT) [18] can retain the global structure and extract high-level visual semantic information for HSI classification [19, 20]. Although the ViT excels at capturing the information contained in spectral-spatial signatures, it fails to make adequate use of spatial information for the division of fixed patches. To address this issue, we develop a novel framework using the well-designed Nested Transformers (NesT) [21] to classify HSI, which realizes the nonlocal feature extraction between pixels and achieves a more refined pixel-wise feature representation.In this paper, the NesT is used for HSI classification for the first time. With the help of the block aggregation module in the NesT, the insufficient feature extraction between patches raised by ViT is mitigated. The experimental results obtained from four hyperspectral datasets show that the NesT is superior to other classical HSI classifiers in terms of accuracy, especially with limited training samples. It can be proved that the NesT has the potential to implement pixel-wise feature extraction.The remainder of this paper is organized as follows. The related work is presented in Section2. Section 3 presents the methodology of the NesT. The details of experimental results are reported in Section 4. In Section 5, relevant conclusions and discussions are presented.


## 2. Related Work

Transformer [22] was proposed for machine translation tasks and has been a standard for natural language processing (NLP) tasks. To search for important information about the input sequence and capture global information, the Transformer architecture transforms one sequence into another through the encoder and decoder based on the self-attention mechanism. These sequences are referred to as “tokens,” which are feature vectors in the original data mapped in the model. The Transformer extracts features of sequence data and assigns attention weights to the raw data mainly by Multilayer Perception (MLP) to build multi-head attention structures. However, the mapping in Transformer considers global information when applied to image data, resulting in a sharp increase on the model scale that brings a heavy burden in calculation and training, so there exists a limitation in image feature extraction [23, 24].Numerous studies have been carried out on the variety of Transformers in the image processing field recently. Bello et al. proposed a local self-attention to replace the convolution layer [25], Wu et al. proposed a multiscale structure to approximate the global self-attention in sparse Transformer [26], Ho et al. introduced the self-attention for different sizes of image blocks to extract the spatial information [27], etc. However, the application results of the above methods were not satisfactory. Until 2021, inspired by the great success of Transformer in the NLP community, ViT was proposed and applied directly to Computer Vision (CV) with the fewest possible modification. When pretrained on the ImageNet-21k dataset and the in-house JFT-300M dataset, ViT approached or outperformed the state-of-the-art model on multiple image recognition benchmarks [28].In previous studies, the tokens in ViT are all fixed scale, which may be unsuitable for vision applications. Hence, the Swin Transformer (Swin) [29] established a hierarchical ViT structure referring to CNN that used multiple feature extractors to expand the receptive field by gradually merging the images with the deepening of network depth. In addition, Swin used a variable-sized attention window to reduce the computation of a single self-attention operation. Although the outputs of each layer were concatenated and then redivided, the Swin Transformer ignores fusing the edge features between patches.Therefore, the NesT was proposed to fuse the spatial features before being divided. The NesT incorporates Swin Transformer features to build a hierarchical ViT model which shares similarities with several pyramid designs. To solve the problem of fixed patch division in ViT, the model carries out simple spatial operations on the outputs of each layer, such as convolution and pooling, to fuse the edge information between patches. The interpatch nonlocal features are extracted and fused after patch segmentation by the block aggregation module, thus effectively improving the ability of spatial feature representation and prediction accuracy. In addition, this additional feature fusion method makes it more efficient to extract data features from the model, which can reduce the width of the network in deeper layers. Therefore, aiming at producing sufficient feature extraction between patches, in the pixel-wise classification task, we introduce a novel workflow for HSI classification based on NesT.


## 3. Methodology

### 3.1. ViT

ViT is a pioneering work in applying Transformer architectures to image classification and has had a profound impact on the design of subsequent visual backbones with its impressive speed and accuracy in image classification. As shown in Figure1, the input data is first split into non-overlapping patches with position embeddings, and each patch is treated as a token. Then, these tokens have constructed a group of learnable sequences and are input into a hierarchical Transformer.Figure 1
Illustration of ViT. An image is split into fixed-size patches, linearly embedded with position embeddings (pos), and the resulting sequence of vectors is fed into a standard Transformer encoder with the standard approach of adding an extra learnable classification token (cls) to the sequence.The Transformer in vision backbones has a standard encoder architecture, but its decoder consists of an MLP head and a classifier. The encoders of the Transformer mine the dependencies between tokens to capture the structural information in the image. The decoders are employed to classify the integrated tokens. In the multi-head self-attention layer of the encoders, a token is mapped to groups of vectors, and each group includes a query (q), a key (k), and a value (v). Next, the dot product of q and k is utilized to measure the dependencies of various tokens. Let T=tii=1Ndenotes the input tokens and S=sii=1Ndenotes the output tokens, where ti∈ℝd1, si∈ℝd1, N is the length of the token sequence, and d1 is the length of the token in each head. The multi-head self-attention layer mines the various dependencies among tokens. Since each token corresponds to a local patch, the dependencies effectively express the structural information of the image. The multi-head self-attention mechanism is calculated by:
(1)qi=Wqti,ki=Wkti,vi=Wvti,(2)si=Wh∑i=1Nsoftmaxqi⋅kid2vi,where qi∈ℝd2, ki∈ℝd2, vi∈ℝd2, Wq∈ℝd1×d2, Wk∈ℝd1×d2, Wv∈ℝd1×d2, Wh∈ℝd1×d2, d2=d1×h, and h is the number of heads. The output tokens are processed by residual connections and layer normalization to pass into the feedforward layer that contains two fully connected operations. Hyperspectral image feature extraction relies on pixels and their neighborhoods, and the window region input to the model includes a variety of objects. On this basis, the tokens further correspond to object-level features, which can express effective structural and discriminative information.


### 3.2. NesT

The NesT is a more effective variant of ViT when integrating global information in the CV. The patch split in ViT directly splits near pixels of HSIs and destroys the correlation of partitioned pixels. The NesT carries out secondary feature extraction that can cover all patches after Transformer to obtain edge information between patches. We introduce the NesT for HSI classification task, as shown in Figure2, consisting of the patch embedding, Transformer encoders, and the block aggregation structure. First, a raw image is split into N non-overlapping patches, added patch and position embeddings, and then fed into the standard Transformer encoders to deeply mine the relationship between pixels. Then, the features between the different patches are extracted through block aggregation. And finally, the label distribution is mapped to the output by the softmax layer.Figure 2
The workflow of HSI classification based on NesT. The patch embedding and Transformer encoders of the model have the same structure as that in ViT. After that, the block aggregation is used to extract the features of different blocks and then do classification.


### 3.3. Hierarchical Structure

Each patch in NesT performs standard Transformer layers to process information independently. Firstly, a hierarchical structure referred to Swin Transformer is built, as shown in Figure3. Secondly, global information is communicated and mixed during the block aggregation step via some simple spatial operations. Performing block aggregation in the image plane can promote information exchange between nearby patches. Taking the 3×3 spatial operation as an example, the output Xl at hierarchy l is unblocked to full image plane Yl∈Rb×H′×W′×d′. Several spatial operations are applied to downsampling feature maps Yl′∈Rb×H′/3×W′/3×d′. Finally, the feature maps are blocked back to Xl+1∈Rb×block/9×n×d′ for hierarchy l+1. It is easy to see the sequence length n always remains the same and the total number of blocks is reduced by a factor of 9 until reduced to 1 at the top hierarchy (i.e., block/9=1). In this way, the receptive field expands gradually by constructing the hierarchically nested structure.Figure 3
Illustration of hierarchical structure in NesT. Different colors represent the output results of different patches. Each time the maximum value in the current block is passed, the hierarchical structure is formed.The nested hierarchy with an independent block process in NesT resembles a tree-like structure that each block is encouraged to learn non-overlapping features and be selected by the block aggregation [21]. This unique feature-driven method explores a new way of interpreting models, named the gradient-based class-aware tree-traversal (GradCAT) method. The main idea is to utilize the tree structure to find the most valuable path from a child node to the root node. As shown in Figure 4, the model can be visualized using GradCAT of the NesT for HSI, which shows how the network selectively utilizes the underlying local visual information to make inferences about the overall category of the image.Figure 4
Illustration of the GradCAT for HSI. (a) The schematic graph of the GradCAT. Given an input image, get the Dim dimension feature maps with the size of5×5, in which each node is the activation value of the patch from the original data in the corresponding dimension. (b) A concrete example. We use a single-layer maximum strategy to select and visualize the blocks which have the greatest impact on classification.
(a)(b)


## 3.1. ViT

ViT is a pioneering work in applying Transformer architectures to image classification and has had a profound impact on the design of subsequent visual backbones with its impressive speed and accuracy in image classification. As shown in Figure1, the input data is first split into non-overlapping patches with position embeddings, and each patch is treated as a token. Then, these tokens have constructed a group of learnable sequences and are input into a hierarchical Transformer.Figure 1
Illustration of ViT. An image is split into fixed-size patches, linearly embedded with position embeddings (pos), and the resulting sequence of vectors is fed into a standard Transformer encoder with the standard approach of adding an extra learnable classification token (cls) to the sequence.The Transformer in vision backbones has a standard encoder architecture, but its decoder consists of an MLP head and a classifier. The encoders of the Transformer mine the dependencies between tokens to capture the structural information in the image. The decoders are employed to classify the integrated tokens. In the multi-head self-attention layer of the encoders, a token is mapped to groups of vectors, and each group includes a query (q), a key (k), and a value (v). Next, the dot product of q and k is utilized to measure the dependencies of various tokens. Let T=tii=1Ndenotes the input tokens and S=sii=1Ndenotes the output tokens, where ti∈ℝd1, si∈ℝd1, N is the length of the token sequence, and d1 is the length of the token in each head. The multi-head self-attention layer mines the various dependencies among tokens. Since each token corresponds to a local patch, the dependencies effectively express the structural information of the image. The multi-head self-attention mechanism is calculated by:
(1)qi=Wqti,ki=Wkti,vi=Wvti,(2)si=Wh∑i=1Nsoftmaxqi⋅kid2vi,where qi∈ℝd2, ki∈ℝd2, vi∈ℝd2, Wq∈ℝd1×d2, Wk∈ℝd1×d2, Wv∈ℝd1×d2, Wh∈ℝd1×d2, d2=d1×h, and h is the number of heads. The output tokens are processed by residual connections and layer normalization to pass into the feedforward layer that contains two fully connected operations. Hyperspectral image feature extraction relies on pixels and their neighborhoods, and the window region input to the model includes a variety of objects. On this basis, the tokens further correspond to object-level features, which can express effective structural and discriminative information.


## 3.2. NesT

The NesT is a more effective variant of ViT when integrating global information in the CV. The patch split in ViT directly splits near pixels of HSIs and destroys the correlation of partitioned pixels. The NesT carries out secondary feature extraction that can cover all patches after Transformer to obtain edge information between patches. We introduce the NesT for HSI classification task, as shown in Figure2, consisting of the patch embedding, Transformer encoders, and the block aggregation structure. First, a raw image is split into N non-overlapping patches, added patch and position embeddings, and then fed into the standard Transformer encoders to deeply mine the relationship between pixels. Then, the features between the different patches are extracted through block aggregation. And finally, the label distribution is mapped to the output by the softmax layer.Figure 2
The workflow of HSI classification based on NesT. The patch embedding and Transformer encoders of the model have the same structure as that in ViT. After that, the block aggregation is used to extract the features of different blocks and then do classification.


## 3.3. Hierarchical Structure

Each patch in NesT performs standard Transformer layers to process information independently. Firstly, a hierarchical structure referred to Swin Transformer is built, as shown in Figure3. Secondly, global information is communicated and mixed during the block aggregation step via some simple spatial operations. Performing block aggregation in the image plane can promote information exchange between nearby patches. Taking the 3×3 spatial operation as an example, the output Xl at hierarchy l is unblocked to full image plane Yl∈Rb×H′×W′×d′. Several spatial operations are applied to downsampling feature maps Yl′∈Rb×H′/3×W′/3×d′. Finally, the feature maps are blocked back to Xl+1∈Rb×block/9×n×d′ for hierarchy l+1. It is easy to see the sequence length n always remains the same and the total number of blocks is reduced by a factor of 9 until reduced to 1 at the top hierarchy (i.e., block/9=1). In this way, the receptive field expands gradually by constructing the hierarchically nested structure.Figure 3
Illustration of hierarchical structure in NesT. Different colors represent the output results of different patches. Each time the maximum value in the current block is passed, the hierarchical structure is formed.The nested hierarchy with an independent block process in NesT resembles a tree-like structure that each block is encouraged to learn non-overlapping features and be selected by the block aggregation [21]. This unique feature-driven method explores a new way of interpreting models, named the gradient-based class-aware tree-traversal (GradCAT) method. The main idea is to utilize the tree structure to find the most valuable path from a child node to the root node. As shown in Figure 4, the model can be visualized using GradCAT of the NesT for HSI, which shows how the network selectively utilizes the underlying local visual information to make inferences about the overall category of the image.Figure 4
Illustration of the GradCAT for HSI. (a) The schematic graph of the GradCAT. Given an input image, get the Dim dimension feature maps with the size of5×5, in which each node is the activation value of the patch from the original data in the corresponding dimension. (b) A concrete example. We use a single-layer maximum strategy to select and visualize the blocks which have the greatest impact on classification.
(a)(b)


## 4. Experiment Results

### 4.1. Datasets

The NesT can efficiently extract features suitable for more complex HSI. In this paper, four public HSI datasets including the Salinas, Indian Pines, Tea Farm, and Xiongan New Area in Matiwan Village (Xiongan) are used to verify the performance of NesT and compared with other deep learning methods. The detailed information of the four datasets is shown in Table1.Table 1
HSI dataset detailed information.
DatasetSalinasIndian PinesTea FarmXionganData size512×217145×145348×5123750×1580Bands22420080256Spatial resolution (m)3.7202.250.5Spectral range (μm)0.4~2.50.4~2.50.417~0.8550.4~1Number of categories16161020The Salinas dataset is collected by an airborne visible/infrared imaging spectrometer (AVIRIS) from the Salinas Valley, California, USA in Figure5(a), while the spectral range is 0.4~2.5 μm with 204 effective bands after removing the bands with severe water absorption and a spatial resolution of 3.7 m. This dataset includes 16 regular fields of different crops.Figure 5
The four HSI datasets with false color images and the classification legends: (a) Salinas; (b) Indian Pines; (c) Tea Farm; (d) Xiongan.
(a)(b)(c)(d)The Indian Pines dataset is also gathered by AVIRIS to perform spectral imaging of a pine forest in Indiana, USA; see Figure5(b). The spectral range is 0.4~2.5 μm with 200 effective retained bands after removing the bands with severe water absorption. It has a spatial resolution of 20 m and includes 16 feature categories.The Tea Farm dataset is captured from the tea planting base in Jiangsu province of China, collected by a push-broom hyperspectral imager (PHI). The spectral range is 0.417~0.855μm with 80 bands. As shown in Figure 5(c), there are 10 types in the labeled categories with a spatial resolution of 2.25 m.In Figure5(d), the Xiongan dataset is an aerial hyperspectral remote sensing image of Matiwan Village in Xiongan New Area of China, which is acquired using the visible and near-infrared imaging spectrometer designed by the Shanghai Institute of Technical Physics, Chinese Academy of Sciences. The spectral range is 0.4~1 μm with 256 bands, and the spatial resolution is 0.5 m. The land cover types labeled here are 20, which are mainly cash crops.


### 4.2. Experiment Setup

In this paper, some experiments are taken to evaluate the performance of NesT and compared with three machine learning classifiers (e.g., LR, SVM, and Random Forest (RF) [31]) and three deep learning classifiers (e.g., CNN, ViT, and Swin Transformer). The typical convolutional structure in the CNN classifier is to extract spatial features through pixel-by-pixel translation and can stack convolution kernels to extract nonlinear features. The application of the ViT classifier shows the performance of combining patch split with Transformer on HSI. The Swin Transformer is used to analyze the impact of different attention window sizes on the classification. The block aggregation is crucial for feature extraction and classification in the NesT, which uses an aggregation nesting approach to achieve feature fusion between patches and adds GradCAT to significantly improve the classification accuracy, compared with the above three models. Furthermore, due to the large number of spectral bands and the existence of information redundancy, principal component analysis (PCA) is performed to reduce the dimensions of the four datasets, and eight principal components are selected to replace the original data according to the variance contribution. Each dataset constructs a training set by extracting the pixels in the 15×15 neighborhood around the sampling points.The NesT is applied to the classification of HSI. We set the patch size as3×3 and used one layer of the original NesT due to the high correlation of pixels and the small scale of features in HSI. A fully connected layer was used to map the features extracted by block aggregation to the category dimension after fusing the features between patches. The detailed parameters of NesT and the comparison models in this paper are shown in Table 2. Finally, we set up Stochastic Gradient Descent (SGD) as the optimizer and trained at 350 iterations with a learning rate of 0.01 and a batch size of 128.Table 2
Parameters of deep learning models.
ModelWindow sizeDimensionDepthHeadMLP dimShifted window sizeGradCATCNN3,3,3,364,256,128,128,10245////ViT312865128//Swin3256872566/NesT325687256/MaxConsidering that Transformer is easy to overfit on limited data, we sampled different percentages of labeled data as the training set to explore the impact on classification and determine the sampling rate for the following experiment. The sampling rate is set as 1%, 3%, and 5%, respectively, of the labeled data for model training, and the accuracy is obtained as shown in Table3. The accuracies of all methods are significantly improved with the increase of sampling rate, which improved most from 1% to 3% and achieved top accuracy at 5%. In the subsequent experiments, we set the sampling rate as 5% to avoid the underfitting caused by insufficient random sampling.Table 3
Classification accuracy of different sampling rates for each dataset (%).
SalinasSampling rate1%3%5%CNN94.3496.5798.93ViT92.3795.6898.66Swin95.0697.7398.76NesT98.5599.4299.61Indian PinesSampling rate1%3%5%CNN62.1475.9286.75ViT71.9082.5190.20Swin64.8872.8775.71NesT69.7484.7290.37Tea FarmSampling rate1%3%5%CNN90.0696.0298.39ViT91.4995.3697.29Swin94.0695.4996.18NesT95.8198.5299.25XionganSampling rate1%3%5%CNN98.4598.9399.70ViT98.3299.4299.73Swin99.2699.7999.90NesT98.3199.5299.80


### 4.3. Investigation of the Model Feature Extraction Performances

The main goal of the t-SNE (t-distributed stochastic neighbor embedding) [30] is to transform the high-dimensional data into low dimensional data for visualization. The t-SNE calculates the similarity among all samples according to the T distribution and then reduces the dimension to project the data randomly to the 2D space.The complexity of the raw data and the features extracted by the model is analyzed before evaluating the classification performance. The t-SNE is used to realize high-dimensional data visualization, as shown in Figures6–9. The following observations are obtained by the analysis of the data distribution after the t-SNE visualization:
(1)
The results of the Salinas dataset indicate that most ground objects are linearly separable since some categories can be easily distinguished from others, while the misjudged categories need to be distinguished by nonlinear mapping via deep learning models(2)
The Indian Pines dataset has the lowest spatial resolution (20 m) compared to the other experimental datasets and exists a more serious phenomenon of mixed pixels which indicates the most nonlinear features within it so that the original data has difficulty in distinguishing(3)
The projection distribution of the Tea Farm dataset with a similar spatial resolution to the Salinas that the primary categories can be distinguished linearly which just have a small scale of hard-to-distinguish categories, requiring the feature extraction and classification by employing deep learning models(4)
The Xiongan dataset has the highest spatial resolution (0.5 m), which results in fewer features of ground objects contained in each pixel, and has the more severe phenomena of “same material, different spectra” and “different material, same spectra,” which made linear classifiers difficult to distinguishFigure 6
The t-SNE visualization results of feature extraction via deep learning methods on Salinas: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)Figure 7
The t-SNE visualization results of feature extraction via deep learning methods on Indian Pines: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)Figure 8
The t-SNE visualization results of feature extraction via deep learning methods on Tea Farm: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)Figure 9
The t-SNE visualization results of feature extraction via deep learning methods on Xiongan: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)The feature extraction performance from different datasets by deep learning models is visualized using t-SNE (see Figures6–9). The CNN has a weaker capability of feature extraction which shows a cross-aggregation state of features in various categories and is hard to classify. The ViT can correctly classify some categories which are misjudged in CNN by extracting separable features through patch embedding and Transformer operations and shows its better data generalizability on different HSI datasets than CNN. The extracted feature by Swin of each category which utilizes the different attention window sizes similarly to the convolutional structure also shows an aggregated distribution phenomenon. The NesT realizes the communication of feature information between patches by adding a block aggregation module, so it has a stronger ability to distinguish easily misjudged categories, which is consistent with results obtained in Figures 6–9.


### 4.4. Results

Tables4–7 show the classification accuracy per model. The results indicate that the classification accuracy of machine learning methods is lower than that of deep learning methods, mainly because machine learning methods cannot combine the spatial features in HSI data, resulting in low overall recognition accuracy.Table 4
The classification accuracy of Salinas (%).
CategoryLRSVMRFCNNViTSwinNesTC0198.39100.0099.74100.00100.00100.00100.00C0296.8598.5499.30100.00100.0099.97100.00C0392.9490.9094.58100.0098.5598.75100.00C0498.3797.6598.36100.0099.7798.9799.13C0593.4297.2298.7298.8997.8098.8499.36C0699.77100.0099.97100.0099.9299.6299.77C0798.8697.4599.4999.94100.0099.94100.00C0873.3069.0278.4197.7897.8698.0399.18C0998.9098.4498.3899.6499.6499.80100.00C1091.0186.3591.7199.5999.6399.8199.93C1187.0188.9393.1499.5298.7099.25100.00C1292.4395.3396.04100.00100.0099.7999.94C1395.5992.8695.9799.89100.0099.89100.00C1496.2397.5798.32100.0099.4399.1699.71C1570.1678.9877.2096.5595.4195.6598.96C1698.3598.8998.9999.72100.00100.00100.00Overall accuracy88.1387.6991.0198.9398.6698.7699.61Table 5
The classification accuracy of Indian Pines (%).
CategoryLRSVMRFCNNViTSwinNesTC0135.8317.8536.5083.6373.0152.4364.78C0260.8956.7264.2073.4078.4757.3183.72C0370.4393.4177.4584.8297.9483.1593.93C0448.100.0069.5294.2196.65100.0094.40C0559.2869.7490.9595.7288.3684.5392.91C0672.6067.3180.9593.0798.3197.1991.20C0735.1324.0052.8390.3272.9738.8867.56C0896.7396.9699.7495.7898.7695.9598.15C0939.130.0044.4462.50100.0064.5180.92C1062.1748.8771.3085.2790.8573.8895.46C1156.8150.1069.6890.6491.7673.2492.35C1228.9426.6662.9277.4080.6947.4972.48C1381.0983.4091.4296.5197.1199.4193.08C1476.4872.6287.7598.9697.7691.8997.54C1558.5583.3372.0293.0085.6480.2486.44C1678.2695.7883.7888.5776.8541.1572.58Overall accuracy63.6758.3574.6886.7590.2075.7190.37Table 6
The classification accuracy of Tea Farm (%).
CategoryLRSVMRFCNNViTSwinNesTC0186.6981.9686.1199.4598.7698.9199.43C0250.1970.7579.7696.2589.7691.8097.82C0393.4294.0194.9498.8096.1196.3899.55C0459.780.0087.2699.4998.61100.00100.00C0596.6596.3696.5699.1298.9098.2999.72C0697.4499.5893.5398.5197.2194.1999.29C0789.1692.5594.1092.4891.6984.7698.57C0857.5059.0469.0085.1272.6673.4794.87C0999.7399.9099.8699.9099.83100.00100.00C1094.4993.6896.4997.2589.9488.4298.77Overall accuracy91.7592.1793.5498.3997.2996.1899.25Table 7
The classification accuracy of Xiongan (%).
CategoryLRSVMRFCNNViTSwinNesTC0133.3772.1287.5699.6099.7399.8499.94C0244.0867.8584.1199.8599.7799.9699.96C030.0076.3787.9698.8299.6099.6199.98C0491.9399.3699.6099.9899.9899.9899.99C0551.9566.6681.6299.8499.7599.8099.91C0645.2773.1287.3899.8599.8699.8999.98C070.0093.0594.1399.8199.9699.8599.89C0875.3297.3498.9999.9599.9199.9899.99C0964.0495.2299.3099.8299.9899.9799.99C1088.7095.5797.7899.8799.9799.9999.99C110.000.0086.7895.1698.4699.5599.80C121.6375.1186.7297.7099.3399.3599.53C1349.9164.1981.1299.8599.6899.8099.82C140.0090.3988.0095.9298.9599.2399.66C1551.6766.6481.8898.2299.0799.0299.49C160.0090.0784.4797.7398.0498.2999.28C170.000.0097.1696.2799.0797.0897.41C1847.5362.8981.5199.7199.6699.7699.89C190.0088.1190.8298.8199.6199.8399.81C2098.9596.4296.9399.2999.6999.8399.93Overall accuracy57.3773.6086.5799.7099.7399.8499.86The NesT exhibits the optimal accuracy of 99.61% in the Salinas dataset, 90.37% in the Indian Pines dataset, 99.25% in the Tea Farm dataset, and 99.86% in the Xiongan dataset. In particular, in the Indian Pines dataset, the accuracy of the NesT is significantly better, with an improvement of 3.62%, 0.17%, and 14.66% over the CNN, ViT, and Swin, respectively. The experimental results show the effectiveness and generalization of the NesT for hyperspectral classification.Furthermore, Figures10–13 depict the visualization results and illustrate the machine learning method unable to utilize spatial information, resulting in the severe salt-and-pepper phenomenon in the prediction results. On the other hand, the deep learning methods use the feature extraction method of space spectrum fusion, which has a good regional prediction effect in HSI.Figure 10
Classification results of Salinas: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 11
Classification results of Indian Pines: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 12
Classification results of Tea Farm: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 13
Classification results of Xiongan: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)From the classification results of the Salinas dataset (Figure10 and Table 4), the misjudged pixels by CNN show continuity as a regional distributed block and are relatively scattered in specific categories. As mentioned, multiple stacked filters are used in CNN to extract features that cause the misjudged regions often fragmented in classification. The result indicates that the CNN just extracts fragmentized information without overall cognition formation. Differently, the Transformers in ViT, Swin, and NesT help to extract the deep-level visual semantic information by focusing on the correlation between features and pixel spatial positions, which results in the phenomenon of “salt and pepper,” shown as discretely distributed on misjudged points. The same signatures shown in classification results can also be observed in Figures 11–13.In the process of mining the internal information between pixels, ViT extracts feature by splitting an image into patches that are easily affected by the noise in HSI. The capability of its MLP structure is relatively weak to extract spatial features which shows the “salt and pepper” phenomenon in prediction results. Moreover, there is no information interaction between patches in ViT, resulting in the inability to integrate information across the image. As a result, the model has good performance on the Indian Pines dataset with more spectral features and fewer spatial features, while the classification accuracy is not satisfactory for the Salinas and Tea Farm datasets with more complex spatial structure.The Swin realizes the feature extraction of overall patches by transforming the patch window size and operating shifted window function to improve the accuracy of HSI with high spatial resolution. However, the poor performance of Swin Transformer on the Indian Pines dataset shows that its optimization is relatively little helpful for the phenomena of “same material, different spectra” and “different material, same spectra.”The NesT achieves top accuracy in four experimental HSI datasets that sufficiently prove the effectiveness and generalization of pixel-wise classification tasks, which is in line with the previously discussed. As can be seen from the NesT classification results in Figures10–13 and Tables 4–7, the phenomenon of salt and pepper is the lightest. The block aggregation structure in the NesT can effectively reduce the number of misjudged pixels caused by the interference of adjacent pixels due to strong correlation, making the model suitable for the extraction of pixel-wise features.The deep learning model can calculate the probability of each category corresponding to this sample in the classifier by the softmax function, which cannot be calculated by traditional machine learning models. The softmax function is used to calculate the entropy of the prediction results for each model in the experiments. For the positive samples (i.e., correctly predicted samples), the smaller the entropy, the better the predictive effect of the model, and the less frequently other error categories appeared in the predicted results. On the contrary, for the negative samples (i.e., misjudged samples), the smaller entropy indicates that the model learned more features from misjudged categories and tended to overfit. The performance of the model can be evaluated by the entropy of positive and negative samples assisted. The entropies of the positive and negative samples in Table8 demonstrate that compared with the NesT, the entropies of negative samples of the CNN are too small in the Salinas and Tea Farm datasets. It is indicated that CNN cannot correctly learn the features extracted from misjudged samples, which can be considered as overfitting. The classification performance of ViT and Swin is lower than that of NesT. Therefore, in most cases, calculating the entropies of positive and negative samples classified by the NesT can also reflect its excellent learning ability of distinguishable features. In more detail, the Xiongan dataset has the highest resolution and more dense features among pixels, so finer features are required to be extracted. The NesT obtains the lowest entropy of positive samples, indicating that it learns more inherent features of ground objects so it is more decisive in classifying the category of a pixel. The NesT has the highest entropies of negative samples in the Salinas and Tea Farm datasets, showing a more hesitant attitude towards the classification of misjudged pixels, which indicates that with appropriate fine-tuning, the misjudged pixels will be more likely to be correctly classified.Table 8
The entropies of the positive and negative samples of different methods.
Entropy (positive)SalinasIndian PinesTea FarmXionganCNN0.46050.48070.15720.4478ViT0.43610.44970.43510.4449Swin0.46530.45680.31460.4362NesT0.44460.44100.16320.4358Entropy (negative)SalinasIndian PinesTea FarmXionganCNN0.18740.46370.16080.4437ViT0.31090.42170.44320.4464Swin0.33010.40090.23710.4450NesT0.37700.45800.47710.4425


## 4.1. Datasets

The NesT can efficiently extract features suitable for more complex HSI. In this paper, four public HSI datasets including the Salinas, Indian Pines, Tea Farm, and Xiongan New Area in Matiwan Village (Xiongan) are used to verify the performance of NesT and compared with other deep learning methods. The detailed information of the four datasets is shown in Table1.Table 1
HSI dataset detailed information.
DatasetSalinasIndian PinesTea FarmXionganData size512×217145×145348×5123750×1580Bands22420080256Spatial resolution (m)3.7202.250.5Spectral range (μm)0.4~2.50.4~2.50.417~0.8550.4~1Number of categories16161020The Salinas dataset is collected by an airborne visible/infrared imaging spectrometer (AVIRIS) from the Salinas Valley, California, USA in Figure5(a), while the spectral range is 0.4~2.5 μm with 204 effective bands after removing the bands with severe water absorption and a spatial resolution of 3.7 m. This dataset includes 16 regular fields of different crops.Figure 5
The four HSI datasets with false color images and the classification legends: (a) Salinas; (b) Indian Pines; (c) Tea Farm; (d) Xiongan.
(a)(b)(c)(d)The Indian Pines dataset is also gathered by AVIRIS to perform spectral imaging of a pine forest in Indiana, USA; see Figure5(b). The spectral range is 0.4~2.5 μm with 200 effective retained bands after removing the bands with severe water absorption. It has a spatial resolution of 20 m and includes 16 feature categories.The Tea Farm dataset is captured from the tea planting base in Jiangsu province of China, collected by a push-broom hyperspectral imager (PHI). The spectral range is 0.417~0.855μm with 80 bands. As shown in Figure 5(c), there are 10 types in the labeled categories with a spatial resolution of 2.25 m.In Figure5(d), the Xiongan dataset is an aerial hyperspectral remote sensing image of Matiwan Village in Xiongan New Area of China, which is acquired using the visible and near-infrared imaging spectrometer designed by the Shanghai Institute of Technical Physics, Chinese Academy of Sciences. The spectral range is 0.4~1 μm with 256 bands, and the spatial resolution is 0.5 m. The land cover types labeled here are 20, which are mainly cash crops.


## 4.2. Experiment Setup

In this paper, some experiments are taken to evaluate the performance of NesT and compared with three machine learning classifiers (e.g., LR, SVM, and Random Forest (RF) [31]) and three deep learning classifiers (e.g., CNN, ViT, and Swin Transformer). The typical convolutional structure in the CNN classifier is to extract spatial features through pixel-by-pixel translation and can stack convolution kernels to extract nonlinear features. The application of the ViT classifier shows the performance of combining patch split with Transformer on HSI. The Swin Transformer is used to analyze the impact of different attention window sizes on the classification. The block aggregation is crucial for feature extraction and classification in the NesT, which uses an aggregation nesting approach to achieve feature fusion between patches and adds GradCAT to significantly improve the classification accuracy, compared with the above three models. Furthermore, due to the large number of spectral bands and the existence of information redundancy, principal component analysis (PCA) is performed to reduce the dimensions of the four datasets, and eight principal components are selected to replace the original data according to the variance contribution. Each dataset constructs a training set by extracting the pixels in the 15×15 neighborhood around the sampling points.The NesT is applied to the classification of HSI. We set the patch size as3×3 and used one layer of the original NesT due to the high correlation of pixels and the small scale of features in HSI. A fully connected layer was used to map the features extracted by block aggregation to the category dimension after fusing the features between patches. The detailed parameters of NesT and the comparison models in this paper are shown in Table 2. Finally, we set up Stochastic Gradient Descent (SGD) as the optimizer and trained at 350 iterations with a learning rate of 0.01 and a batch size of 128.Table 2
Parameters of deep learning models.
ModelWindow sizeDimensionDepthHeadMLP dimShifted window sizeGradCATCNN3,3,3,364,256,128,128,10245////ViT312865128//Swin3256872566/NesT325687256/MaxConsidering that Transformer is easy to overfit on limited data, we sampled different percentages of labeled data as the training set to explore the impact on classification and determine the sampling rate for the following experiment. The sampling rate is set as 1%, 3%, and 5%, respectively, of the labeled data for model training, and the accuracy is obtained as shown in Table3. The accuracies of all methods are significantly improved with the increase of sampling rate, which improved most from 1% to 3% and achieved top accuracy at 5%. In the subsequent experiments, we set the sampling rate as 5% to avoid the underfitting caused by insufficient random sampling.Table 3
Classification accuracy of different sampling rates for each dataset (%).
SalinasSampling rate1%3%5%CNN94.3496.5798.93ViT92.3795.6898.66Swin95.0697.7398.76NesT98.5599.4299.61Indian PinesSampling rate1%3%5%CNN62.1475.9286.75ViT71.9082.5190.20Swin64.8872.8775.71NesT69.7484.7290.37Tea FarmSampling rate1%3%5%CNN90.0696.0298.39ViT91.4995.3697.29Swin94.0695.4996.18NesT95.8198.5299.25XionganSampling rate1%3%5%CNN98.4598.9399.70ViT98.3299.4299.73Swin99.2699.7999.90NesT98.3199.5299.80


## 4.3. Investigation of the Model Feature Extraction Performances

The main goal of the t-SNE (t-distributed stochastic neighbor embedding) [30] is to transform the high-dimensional data into low dimensional data for visualization. The t-SNE calculates the similarity among all samples according to the T distribution and then reduces the dimension to project the data randomly to the 2D space.The complexity of the raw data and the features extracted by the model is analyzed before evaluating the classification performance. The t-SNE is used to realize high-dimensional data visualization, as shown in Figures6–9. The following observations are obtained by the analysis of the data distribution after the t-SNE visualization:
(1)
The results of the Salinas dataset indicate that most ground objects are linearly separable since some categories can be easily distinguished from others, while the misjudged categories need to be distinguished by nonlinear mapping via deep learning models(2)
The Indian Pines dataset has the lowest spatial resolution (20 m) compared to the other experimental datasets and exists a more serious phenomenon of mixed pixels which indicates the most nonlinear features within it so that the original data has difficulty in distinguishing(3)
The projection distribution of the Tea Farm dataset with a similar spatial resolution to the Salinas that the primary categories can be distinguished linearly which just have a small scale of hard-to-distinguish categories, requiring the feature extraction and classification by employing deep learning models(4)
The Xiongan dataset has the highest spatial resolution (0.5 m), which results in fewer features of ground objects contained in each pixel, and has the more severe phenomena of “same material, different spectra” and “different material, same spectra,” which made linear classifiers difficult to distinguishFigure 6
The t-SNE visualization results of feature extraction via deep learning methods on Salinas: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)Figure 7
The t-SNE visualization results of feature extraction via deep learning methods on Indian Pines: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)Figure 8
The t-SNE visualization results of feature extraction via deep learning methods on Tea Farm: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)Figure 9
The t-SNE visualization results of feature extraction via deep learning methods on Xiongan: (a) raw data; (b) CNN; (c) ViT; (d) Swin; (e) NesT.
(a)(b)(c)(d)(e)The feature extraction performance from different datasets by deep learning models is visualized using t-SNE (see Figures6–9). The CNN has a weaker capability of feature extraction which shows a cross-aggregation state of features in various categories and is hard to classify. The ViT can correctly classify some categories which are misjudged in CNN by extracting separable features through patch embedding and Transformer operations and shows its better data generalizability on different HSI datasets than CNN. The extracted feature by Swin of each category which utilizes the different attention window sizes similarly to the convolutional structure also shows an aggregated distribution phenomenon. The NesT realizes the communication of feature information between patches by adding a block aggregation module, so it has a stronger ability to distinguish easily misjudged categories, which is consistent with results obtained in Figures 6–9.


## 4.4. Results

Tables4–7 show the classification accuracy per model. The results indicate that the classification accuracy of machine learning methods is lower than that of deep learning methods, mainly because machine learning methods cannot combine the spatial features in HSI data, resulting in low overall recognition accuracy.Table 4
The classification accuracy of Salinas (%).
CategoryLRSVMRFCNNViTSwinNesTC0198.39100.0099.74100.00100.00100.00100.00C0296.8598.5499.30100.00100.0099.97100.00C0392.9490.9094.58100.0098.5598.75100.00C0498.3797.6598.36100.0099.7798.9799.13C0593.4297.2298.7298.8997.8098.8499.36C0699.77100.0099.97100.0099.9299.6299.77C0798.8697.4599.4999.94100.0099.94100.00C0873.3069.0278.4197.7897.8698.0399.18C0998.9098.4498.3899.6499.6499.80100.00C1091.0186.3591.7199.5999.6399.8199.93C1187.0188.9393.1499.5298.7099.25100.00C1292.4395.3396.04100.00100.0099.7999.94C1395.5992.8695.9799.89100.0099.89100.00C1496.2397.5798.32100.0099.4399.1699.71C1570.1678.9877.2096.5595.4195.6598.96C1698.3598.8998.9999.72100.00100.00100.00Overall accuracy88.1387.6991.0198.9398.6698.7699.61Table 5
The classification accuracy of Indian Pines (%).
CategoryLRSVMRFCNNViTSwinNesTC0135.8317.8536.5083.6373.0152.4364.78C0260.8956.7264.2073.4078.4757.3183.72C0370.4393.4177.4584.8297.9483.1593.93C0448.100.0069.5294.2196.65100.0094.40C0559.2869.7490.9595.7288.3684.5392.91C0672.6067.3180.9593.0798.3197.1991.20C0735.1324.0052.8390.3272.9738.8867.56C0896.7396.9699.7495.7898.7695.9598.15C0939.130.0044.4462.50100.0064.5180.92C1062.1748.8771.3085.2790.8573.8895.46C1156.8150.1069.6890.6491.7673.2492.35C1228.9426.6662.9277.4080.6947.4972.48C1381.0983.4091.4296.5197.1199.4193.08C1476.4872.6287.7598.9697.7691.8997.54C1558.5583.3372.0293.0085.6480.2486.44C1678.2695.7883.7888.5776.8541.1572.58Overall accuracy63.6758.3574.6886.7590.2075.7190.37Table 6
The classification accuracy of Tea Farm (%).
CategoryLRSVMRFCNNViTSwinNesTC0186.6981.9686.1199.4598.7698.9199.43C0250.1970.7579.7696.2589.7691.8097.82C0393.4294.0194.9498.8096.1196.3899.55C0459.780.0087.2699.4998.61100.00100.00C0596.6596.3696.5699.1298.9098.2999.72C0697.4499.5893.5398.5197.2194.1999.29C0789.1692.5594.1092.4891.6984.7698.57C0857.5059.0469.0085.1272.6673.4794.87C0999.7399.9099.8699.9099.83100.00100.00C1094.4993.6896.4997.2589.9488.4298.77Overall accuracy91.7592.1793.5498.3997.2996.1899.25Table 7
The classification accuracy of Xiongan (%).
CategoryLRSVMRFCNNViTSwinNesTC0133.3772.1287.5699.6099.7399.8499.94C0244.0867.8584.1199.8599.7799.9699.96C030.0076.3787.9698.8299.6099.6199.98C0491.9399.3699.6099.9899.9899.9899.99C0551.9566.6681.6299.8499.7599.8099.91C0645.2773.1287.3899.8599.8699.8999.98C070.0093.0594.1399.8199.9699.8599.89C0875.3297.3498.9999.9599.9199.9899.99C0964.0495.2299.3099.8299.9899.9799.99C1088.7095.5797.7899.8799.9799.9999.99C110.000.0086.7895.1698.4699.5599.80C121.6375.1186.7297.7099.3399.3599.53C1349.9164.1981.1299.8599.6899.8099.82C140.0090.3988.0095.9298.9599.2399.66C1551.6766.6481.8898.2299.0799.0299.49C160.0090.0784.4797.7398.0498.2999.28C170.000.0097.1696.2799.0797.0897.41C1847.5362.8981.5199.7199.6699.7699.89C190.0088.1190.8298.8199.6199.8399.81C2098.9596.4296.9399.2999.6999.8399.93Overall accuracy57.3773.6086.5799.7099.7399.8499.86The NesT exhibits the optimal accuracy of 99.61% in the Salinas dataset, 90.37% in the Indian Pines dataset, 99.25% in the Tea Farm dataset, and 99.86% in the Xiongan dataset. In particular, in the Indian Pines dataset, the accuracy of the NesT is significantly better, with an improvement of 3.62%, 0.17%, and 14.66% over the CNN, ViT, and Swin, respectively. The experimental results show the effectiveness and generalization of the NesT for hyperspectral classification.Furthermore, Figures10–13 depict the visualization results and illustrate the machine learning method unable to utilize spatial information, resulting in the severe salt-and-pepper phenomenon in the prediction results. On the other hand, the deep learning methods use the feature extraction method of space spectrum fusion, which has a good regional prediction effect in HSI.Figure 10
Classification results of Salinas: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 11
Classification results of Indian Pines: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 12
Classification results of Tea Farm: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 13
Classification results of Xiongan: (a) groundtruth; (b) LR; (c) SVM; (d) RF; (e) CNN; (f) ViT; (g) Swin; (h) NesT.
(a)(b)(c)(d)(e)(f)(g)(h)From the classification results of the Salinas dataset (Figure10 and Table 4), the misjudged pixels by CNN show continuity as a regional distributed block and are relatively scattered in specific categories. As mentioned, multiple stacked filters are used in CNN to extract features that cause the misjudged regions often fragmented in classification. The result indicates that the CNN just extracts fragmentized information without overall cognition formation. Differently, the Transformers in ViT, Swin, and NesT help to extract the deep-level visual semantic information by focusing on the correlation between features and pixel spatial positions, which results in the phenomenon of “salt and pepper,” shown as discretely distributed on misjudged points. The same signatures shown in classification results can also be observed in Figures 11–13.In the process of mining the internal information between pixels, ViT extracts feature by splitting an image into patches that are easily affected by the noise in HSI. The capability of its MLP structure is relatively weak to extract spatial features which shows the “salt and pepper” phenomenon in prediction results. Moreover, there is no information interaction between patches in ViT, resulting in the inability to integrate information across the image. As a result, the model has good performance on the Indian Pines dataset with more spectral features and fewer spatial features, while the classification accuracy is not satisfactory for the Salinas and Tea Farm datasets with more complex spatial structure.The Swin realizes the feature extraction of overall patches by transforming the patch window size and operating shifted window function to improve the accuracy of HSI with high spatial resolution. However, the poor performance of Swin Transformer on the Indian Pines dataset shows that its optimization is relatively little helpful for the phenomena of “same material, different spectra” and “different material, same spectra.”The NesT achieves top accuracy in four experimental HSI datasets that sufficiently prove the effectiveness and generalization of pixel-wise classification tasks, which is in line with the previously discussed. As can be seen from the NesT classification results in Figures10–13 and Tables 4–7, the phenomenon of salt and pepper is the lightest. The block aggregation structure in the NesT can effectively reduce the number of misjudged pixels caused by the interference of adjacent pixels due to strong correlation, making the model suitable for the extraction of pixel-wise features.The deep learning model can calculate the probability of each category corresponding to this sample in the classifier by the softmax function, which cannot be calculated by traditional machine learning models. The softmax function is used to calculate the entropy of the prediction results for each model in the experiments. For the positive samples (i.e., correctly predicted samples), the smaller the entropy, the better the predictive effect of the model, and the less frequently other error categories appeared in the predicted results. On the contrary, for the negative samples (i.e., misjudged samples), the smaller entropy indicates that the model learned more features from misjudged categories and tended to overfit. The performance of the model can be evaluated by the entropy of positive and negative samples assisted. The entropies of the positive and negative samples in Table8 demonstrate that compared with the NesT, the entropies of negative samples of the CNN are too small in the Salinas and Tea Farm datasets. It is indicated that CNN cannot correctly learn the features extracted from misjudged samples, which can be considered as overfitting. The classification performance of ViT and Swin is lower than that of NesT. Therefore, in most cases, calculating the entropies of positive and negative samples classified by the NesT can also reflect its excellent learning ability of distinguishable features. In more detail, the Xiongan dataset has the highest resolution and more dense features among pixels, so finer features are required to be extracted. The NesT obtains the lowest entropy of positive samples, indicating that it learns more inherent features of ground objects so it is more decisive in classifying the category of a pixel. The NesT has the highest entropies of negative samples in the Salinas and Tea Farm datasets, showing a more hesitant attitude towards the classification of misjudged pixels, which indicates that with appropriate fine-tuning, the misjudged pixels will be more likely to be correctly classified.Table 8
The entropies of the positive and negative samples of different methods.
Entropy (positive)SalinasIndian PinesTea FarmXionganCNN0.46050.48070.15720.4478ViT0.43610.44970.43510.4449Swin0.46530.45680.31460.4362NesT0.44460.44100.16320.4358Entropy (negative)SalinasIndian PinesTea FarmXionganCNN0.18740.46370.16080.4437ViT0.31090.42170.44320.4464Swin0.33010.40090.23710.4450NesT0.37700.45800.47710.4425


## 5. Conclusion

In this paper, we propose a novel workflow based on the NesT for HSI classification. With the help of the block aggregation module in the NesT, the problem of fixed patch division raised by ViT is mitigated, which causes sufficient feature extraction between patches. The applicability and generalization of the NesT for HSI classification are explored for the first time. We conduct experiments on four public HSI datasets to investigate the performance of the NesT, compared with other classical hyperspectral classification methods. The experimental results show that deep learning methods can extract HSI spectral–spatial features and improve the classification accuracy of the model compared with machine learning methods. For the deep learning models, NesT algorithm achieves the best performance in feature extraction and classification when all scenarios are considered. The NesT has huge potential as a candidate for hyperspectral classification with a better model visualization. Our work enriches the usage scenarios of the NesT and opens a new window for HSI classification. However, NesT also contains limitations, such as the requirement of high computing costs and training instances. In the future, we will conduct extensive tests analyzing the performance-complexity tradeoff of the NesT and try to apply it to the semantic segmentation and reconstruction of HSI.



---

*Source: 6785966-2022-07-25.xml*