# Multiscale Dense Cross-Attention Mechanism with Covariance Pooling for Hyperspectral Image Scene Classification

Authors: Runmin Liu; Xin Ning; Weiwei Cai; Guangjun Li
Journal: Mobile Information Systems
 (2021)

---

## Abstract

In recent years, learning algorithms based on deep convolution frameworks have gradually become the research hotspots in hyperspectral image classification tasks. However, in the classification process, high-dimensionality problems with large amounts of data and feature redundancy with interspectral correlation of hyperspectral images have not been solved efficiently. Therefore, this paper investigates data dimensionality reduction and feature extraction and proposes a novel multiscale dense cross-attention mechanism algorithm with covariance pooling (MDCA-CP) for hyperspectral image scene classification. The multisize convolution module can detect subtle changes in the hyperspectral images’ spatial and spectral dimensions between the pixels in the local areas and are suitable for extracting hyperspectral data with complex and diverse types of structures. For traditional algorithms that assign attention weights in a one-way manner, thus leading to the loss of feature information, the dense cross-attention mechanism proposed in this study can jointly distribute the attention weights horizontally and vertically to efficiently capture the most representative features. In addition, this study also uses covariance pooling to further extract the features of hyperspectral images from the second order. Experiments have been conducted on three well-known hyperspectral datasets, and the results thus obtained show that the MDCA-CP algorithm is superior compared to the other well-known methods.


---

## Body

## 1. Introduction

Hyperspectral remote sensing technology can generate a set of images with rich information. It can significantly improve data analysis quality, with significant improvements in the method’s detail, reliability, and credibility. In general, because different categories of the same feature target exhibit different optical behavior and shown obvious differences in the band space, the feature type’s pixel level can be used for identification and classification. This technology is nowadays being used in agriculture [1–3], environmental remote sensing [4–6], physics [7, 8], ocean [9–11], and other fields, among which the research on hyperspectral image (HSI) classification is extremely important. The internal information of an image contains various types of features. It is precisely because of the difference in the various forms of features inside an image that they can be extracted for analysis. Finally, the method of distinguishing the target into different categories is called image classification. HSI classification enables identifying different targets of the ground features based on the obtained remote sensing information. In addition, it can also analyze the inherent laws of the ground features during the classification process. As HSI acquisition costs are relatively high, this will lead to an insufficient number of training samples. Further, the image itself has a relatively high spectral dimension and a relatively large amount of data, which increases its complexity.In the early research, spectral information for classification was a popular procedure [12–14]. The methods of feature selection and dimensionality reduction [15, 16] were often used to ease the spectral dimension’s high dimensionality. With advances in this research topic, the complex space and spectral feature distribution of HSI have become the main problems that plague their classification. Many researchers have chosen to add local spatial connections to improve the models [16, 17] and have also achieved positive results to a certain extent. However, these methods are primarily based on manual and shallow models, which rely heavily on expert knowledge and have poor generalization ability, making it difficult to extract the representative discriminating features.In recent years, deep learning models have been fruitful and have emerged as a new research hotspot among the different methods used for HSI classification. Although deep learning methods can obtain deep-level feature representations from the original HSIs, detection and recognition usually rely on many training samples. However, it is time-consuming and labor-intensive to label the data used for HSI classification and to use it for training. The annotation data are very limited. In view of the abovementioned problems, maintaining deep learning methods with fewer training samples is still a significant challenge. In 2014, Chen et al. [18] proposed a deep learning framework that combines spatial and spectral features. The deep learning framework combines the principal component analysis (PCA) and deep learning architecture employing stacked autoencoders (SAE) to obtain the deep features using logistic regression (LR) results in classification (SAE-LR). Although the SAE-LR deep learning method has shown great potential in HSI classification, the autoencoder model usually flattens the local image patches into vectors and then feeds them into the model. However, this method destroys the image’s two-dimensional structure, resulting in the loss of spatial information and the training time is also insufficient. In 2015, Makantasis et al. [19] used convolutional neural networks (CNNs) to develop a deep-supervised HSI classification algorithm. Their algorithm uses random PCA to reduce the original input data’s dimensionality, a CNN to extract the deep-level features, and multilayer perceptron (MLP) for classification. In 2016, Zhao and Du proposed the spectral-spatial feature classification (SSFC) method. The framework uses a balanced local discriminant embedding (BLDE) algorithm to extract the spectral features based on the classification’s multifeature classifiers.In addition, three-dimensional spectral-spatial feature extraction methods have also been developed to discover useful but difficult-to-extract information from HSIs. Tsai et al. [20] proposed a three-dimensional gray-level co-occurrence feature extraction method for HSI classification based on the traditional two-dimensional morphological operators, and Weeks et al. [21] proposed a three-dimensional discrete wavelet transform, which is a series of one-dimensional wavelet transforms along the three-direction sequence of HSI processing and classification. Feature extraction from HSIs is important for improving the classification performance of the method. The related model of deep learning provides a new solution to the problems in HSI classification. However, applying deep learning to such a classification task also faces some difficulties.(1)
The high-dimensional problem of HSIs is different from ordinary natural images. HSI data present a three-dimensional structure. The spectral information is very rich, and the spatial information is relatively small. The application of deep learning for the classification of HSIs is the first step to designing a network structure suitable for HSI structure, which can effectively use the HSIs to enrich the spectral information and the limited spatial information.(2)
In the HSI data, there is often a serious imbalance between the number of classes and the number of classes. If the weight constraints of various samples are not imposed, the trained network’s robustness is low.Based on the above limitations, a novel multiscale dense cross-attention mechanism algorithm with covariance pooling (MDCA-CP) has been proposed in this work for HSI scene classification. Multiscale convolution can detect the spatial dimensions of HSIs. The subtle changes between the pixels in the local areas in the spectral dimension can be applied to the feature extraction of the hyperspectral data of complex and diverse types and structures. Traditional algorithms only assign attention weights in a one-way manner, resulting in the loss of feature information. The dense cross-attention mechanism proposed in this work can simultaneously distribute the attention weights horizontally and vertically to efficiently fetch the most representative hyperspectral data. In addition, covariance pooling has been used in this study to further extract the second-order features of HSIs. Experiments have been conducted on three well-known hyperspectral datasets, and the results thus obtained prove the effectiveness of the MDCA-CP algorithm.The main contributions of this work are as follows:(1)
A novel dense cross-attention (horizontal and vertical) mechanism algorithm has been proposed, and weight addition and maximum weight strategies have been constructed, which can mine more representative features in the hyperspectral data.(2)
A novel attention-guided covariance pooling method has been proposed to make full use of the second-order information in the hyperspectral data feature map. It allows the neural network to learn more representative features when dealing with remote sensing scene classification problems.(3)
Many experiments and ablation studies have been conducted on three public hyperspectral datasets of Indian Pines, Salinas, and Pavia University, and visual analysis of the results has been done, proving its effectiveness superiority of the MDCA-CP algorithm.The rest of the paper has been organized as follows: Section2 briefly describes other works related to this article. Section 3 introduces the principle and implementation details of the MDCA-CP algorithm. Section 4 presents the experimental results and visual analysis. Section 5 provides a summary of the conclusions based on this research.


## 2. Related Work

In recent years, deep learning has been mainly based on multilayer neural networks that infinitely approximate the nonlinear functions by using three or more neural networks to learn large amounts of abstract feature information in the image. Because it is considerably difficult to obtain the labels for HSIs, the use of deep learning methods to achieve better classification results while using a small number of training samples has always been the focus of research in the field of hyperspectral remote sensing. The classic neural network models based on deep learning include the autoencoder (AE) [22], stacked autoencoder (SAE) [23], and restricted Boltzmann machine (RBM) [24, 25]. Although these methods have relatively good classification performance, scholars have designed deep neural network models based on CNNs and others due to many model parameters and shallow image features. The methods are combined and applied to the classification of HSIs and provide superior feature extraction effect and high classification performance.Chen et al. [18] proposed deep feature extraction and HSI classification method based on CNNs. The CNN is used for extracting the deep features from the HSIs that are nonlinear, discriminative, and invariant. A combination of the CNN and a regularization method is used for extracting the spatial-spectral features of HSIs for classification. Xu et al. [26] proposed a multisource remote sensing data classification method based on CNN, which mainly uses two channels. The CNN extracts the spatial-spectral joint features of the HSI, fuses them with the features of other sources of remote sensing data, and classifies them. Luo et al. [27] proposed a new HSI classification method based on CNN in which the spatial-spectral features from the target pixel and its neighboring pixels are extracted. The convolution operations are performed on them, and the convolution results are superimposed into a two-dimensional matrix as the input of the standard CNN. Finally, the XGBoost classification model, proposed by Wang et al. [28], is an HSI classification method based on a combination of the random forest technique and CNN. It regards the CNN as an individual classifier that is used for extracting the discriminating features of HSIs. The random forest technique involves a random selection of the extracted features and training samples to establish a multiclassification system for performing a classification task.The abovementioned models based on CNNs are all constructed with standard convolutional layers and applied to HSI classification. In order to enable the models to gain the huge advantages of the standard convolutional layer while saving a considerable amount of computing resources, some scholars have introduced the attention mechanism into the classification models. Mei et al. [29] proposed a spectral-spatial attention network for HSI classification. This method can learn the internal spectral correlation in the continuous spectrum, and the attention model focuses on the neighboring pixels in the spatial dimension. Experimental results show that this method can make full use of spectral information. In order to extract the spectral and spatial features, Ma et al. [30] proposed a dual-branch multiattention mechanism network (DBMA) for HSI classification. The network has two branches that extract the spectral and spatial features, respectively, reducing the two features’ interference. In addition, because of the two branches’ different characteristics, two types of attention mechanisms are applied in the two branches, respectively, which ensures more distinctive spectral and spatial features. Excellent results have been achieved using this method. The abovementioned research studies show that the attention mechanism [31, 32] is extremely effective in hyperspectral data classification tasks and provides outstanding results.


## 3. Methodology

Figure1 shows a schematic of the overall architecture of our MDCA-CP algorithm. First, locality preserving projections (LPPs) are used for reducing the dimensionality of the HSI to eliminate redundant information and reduce computational costs. Second, the reduced-dimensional features are inputted into our MDCA module, and the second-order features are extracted through CP. Finally, the SoftMax function is used for predicting each pixel to achieve the classification of the hyperspectral data.Figure 1
Schematic showing our MDCA-CP algorithm’s overall architecture.⊗ represents the feature fusion operation, whereas ⊕ represents the feature concatenation operation.


### 3.1. Locality Preserving Projections

The spectral image of each pixel of the HSI has multiple endmember spectral images at the same time. In a low-dimensional subspace, the high-dimensional data can be projected onto this low-latitude subspace when the HSI is classified. The contained pixels represent a single endmember, which represents a characteristic of a ground feature. The local preserving dimensionality reduction method proposed in this study can find a linear mapM. The adjacent pixels of the original HSI can obtain relatively close projection space distances, therefore achieving effective retention of the relevant information in the local neighborhood and efficiently protecting the diversified local structure of the original HSI. Assuming that the original data’s training sample is xi, the class label xj is the number of sample categories, xth is the total number of training samples, and the training samples are xi and xj. Then, the close relationship equation between xi and xj can be expressed as follows:(1)Aij=exp−xi−xj2γi−γj,where γi=xi−xim represents the local scale of the sample xi and xim represents the mth adjacent sample of the pixel xi. The local interclass scatter matrices Llb and Llw in the local preservation dimensionality reduction are defined as(2)Llb=12∑i,j=1nWi,jlbxi−xjxi−xjT,Llw=∑i,j=1nWi,jlwxi−xjxi−xjT.BothLlb and Llw are n × n-dimensional square matrices.Equation (1) provides the weights of the adjacent pixels of the same kind, and the interclass scatter matrices Llb and Llw hardly affect the same nonadjacent pixels. By locally spreading the matrix, the maximized Fisher ratio is given by(3)LlbT=δLlwT.The transformation matrix obtained by deformation calculation is(4)TLFDA=argmaxtrT∈Rd×rTLFDATLlwTLFDA−1TLFDATLlbTLFDA.LPPs reduce the adjacent data of the same kind via the transformation matrix,T. They effectively separate the adjacent data of different types while retaining the data’s local characteristics. Using the local retention and dimensionality reduction to preprocess the original HSI, the redundant information is eliminated. The distribution of similar classes is made more compact, the noise is reduced, and the classification accuracy is improved.


### 3.2. Multiscale Convolution

In this work, the use of a multiscale convolution kernel has two major advantages:The biggest advantage of multiscale convolution kernels is that different sizes of the convolution kernels can extract features of different scales from the HSIs. Thus, the filter can extract and learn richer information from them.When the CNN training model is used, training is achieved by learning the parameters of the filter (weights and offsets), i.e., by continuously learning the parameters of the filter to achieve an optimal value that is closest to the label. In this study, a multiscale convolution kernel has been used to enable every convolutional layer to have a variety of filters, thus achieving diversification of the weight and bias learning and a complete and effective extraction and learning of the useful information from the HSI.In computer vision models, multiscale inference methods are usually used for obtaining the best results. Generally, the finer details are efficiently predicted at larger sizes, the larger objects are efficiently predicted at smaller sizes, and the receiving field of the network can accurately understand the scene at smaller sizes. In contrast to the traditional multiscale structure, a novel multiscale dense convolution model (as shown in Figure2) has been proposed in this study. In particular, we apply the convolution process to extract the features in four sizes: 11 × 11, 7 × 7, 5 × 5, and 3 × 3. The corresponding calculation formulae are as follows:(5)Y1=ϕ∑i=0n1wij1∗xi1+bj1,Y2=addϕ∑i=0n1wij1∗xi1+bj1,ϕ∑i=0n2wij2∗xi2+bj2,Y3=ϕ∑i=0n3wij3∗xi3+bj3,Y4=addϕ∑i=0n1wij1∗xi1+bj1,ϕ∑i=0n2wij2∗xi2+bj2,ϕ∑i=0n3wij3∗xi3+bj3,Y5=addϕ∑i=0n1wij1∗xi1+bj1,ϕ∑i=0n2wij2∗xi2+bj2,ϕ∑i=0n3wij3∗xi3+bj3,ϕ∑i=0n4wij4∗xi4+bj4,where xi1, xi2, xi3, xi4, wij1, wij2, wij3, and wij4 represent the neurons and weights from the kernel of the multiscale convolutional layer and n represents the number of filters in CONV_i. As the output of the convolutional layer, Y is fed to the cross-attention-guided CP module.Figure 2
Schematic showing the multiscale layer that includes four different scale convolutions.Y1, Y2, Y3, Y4, and Y5 are the feature maps from the different scale convolutions.


### 3.3. Cross-Attention Mechanism

#### 3.3.1. Attention Mechanism

In essence, deep learning’s attention mechanism is similar to the visual attention mechanism humans use when making choices. Both aim to filter out the key information, which is more helpful to the task they are dealing with, from a large amount of information, and suppress the useless information that is not relevant to the task at hand. The attention mechanism is prominent in two aspects: on the one hand, it can select the locally important information that is required to be focused on the overall input by itself; on the other hand, it can reasonably allocate a small amount of important information to the critical task objectives via computing resources. Because these two aspects, which enable the current attention mechanism network to be widely applied in image recognition, stand out, the attention mechanism can strengthen the characteristics of the local information. Further, the attention area position will change depending on the mission objectives; by understanding the target’s local information, the most useful information corresponding to the target can be put to good use. Figure3 shows the framework of the attention mechanism network. The output of the decoder part can be expressed as follows:(6)St=fst−1,yt−1,Ct,where St is the state output of the decoder at a time t, St−1 is the state output of the decoder at a time t−1, yt−1 is the label at a time t−1, and f is the dense layer.(7)Ct=∑j=1Txatjhj,where Ct is the output of the next state, hj is the output of the jth input in the decoder, and atj is the attention weight.(8)atj=expetj∑k=1Txexpetk,etj=gst−1,hj=v×tanhW×hj+u×St−1+b,where atj represents the degree of alignment between the current decoder and the jth input, which is the attention weight. g is used for calculating the relationship score between St−1 and hj. It can be seen that if the score is higher, the attention distribution is focused on the input.Figure 3
Schematic of the attention mechanism model.


#### 3.3.2. Cross-Attention

Considering that the traditional attention mechanism assigns attention weight from only a single direction, it often leads to the loss of feature information. This study proposes a cross-attention mechanism, which constructs two novel weight distribution strategies: weight addition and maximization strategies. First, the cross-attention mechanism calculates the feature weight coefficients along with the horizontal and vertical directions. These two weight coefficients are added in order to enhance the feature, and the largest weight coefficient is obtained by applying the maximization strategy. Finally, the output of the two strategies is obtained via fusion. Figure4 shows a schematic diagram of the cross-attention mechanism.Figure 4
Schematic of the cross-attention mechanism.The equations for calculating the addition and maximization of weights are as follows:(9)Stj=∑j=1Txexpetj∑k=1Txexpetkhj,add=S1+S2,max=S1,S2,Cross_att=concatenateS1,S2,add,max,where S1 and S2 represent the weight coefficients of the horizontal and vertical attention mechanisms, respectively, add represents the addition of the weight coefficients, and max represents the maximization operation of the weight coefficients.


### 3.4. Covariance Pooling

The convolutional layer of a traditional CNN uses max/average pooling, and the fully connected layer can only capture the first-order information. Although the rectified linear unit (ReLU) introduces nonlinearity, it is only limited to the single-pixel level. We believe that the covariance matrix pooling (as shown in Figure5) can extract the features from HSIs with higher efficiency than the first-order statistics. If it is a set of features, its covariance matrix is(10)CP=1n−1∑i=1nfi−f¯fi−f¯T,where f¯=1/n∑i=1nfi. Only when the number of linear independent components in f1,f2,…,fn is greater than d, the resulting matrix is a symmetric positive definite (SPD) matrix. The covariance matrix must be SPD to use the SPD manifold network’s geometric structure to maintain the layer. However, even if the matrix is only positive semidefinite, it can be normalized by adding a multiple of the trace to the diagonal term of the covariance matrix as follows:(11)CP+=CP+λtraceCPI,where λ is the regular parameter and I is the identity matrix.Figure 5
Illustration of the concept of covariance pooling.


### 3.5. Feature Fusion

The multiscale convolution’s output features were spliced, and the CP that is guided by the cross-attention mechanism was used for obtaining the final deep features. The calculation equation for the same is written as follows:(12)O=addCP+Y1,CP+Cross_attY2,CP+Y3,CP+Cross_attY4,CP+Cross_attY5,where O represents the output of the CP module guided by the multiscale cross-attention mechanism.


## 3.1. Locality Preserving Projections

The spectral image of each pixel of the HSI has multiple endmember spectral images at the same time. In a low-dimensional subspace, the high-dimensional data can be projected onto this low-latitude subspace when the HSI is classified. The contained pixels represent a single endmember, which represents a characteristic of a ground feature. The local preserving dimensionality reduction method proposed in this study can find a linear mapM. The adjacent pixels of the original HSI can obtain relatively close projection space distances, therefore achieving effective retention of the relevant information in the local neighborhood and efficiently protecting the diversified local structure of the original HSI. Assuming that the original data’s training sample is xi, the class label xj is the number of sample categories, xth is the total number of training samples, and the training samples are xi and xj. Then, the close relationship equation between xi and xj can be expressed as follows:(1)Aij=exp−xi−xj2γi−γj,where γi=xi−xim represents the local scale of the sample xi and xim represents the mth adjacent sample of the pixel xi. The local interclass scatter matrices Llb and Llw in the local preservation dimensionality reduction are defined as(2)Llb=12∑i,j=1nWi,jlbxi−xjxi−xjT,Llw=∑i,j=1nWi,jlwxi−xjxi−xjT.BothLlb and Llw are n × n-dimensional square matrices.Equation (1) provides the weights of the adjacent pixels of the same kind, and the interclass scatter matrices Llb and Llw hardly affect the same nonadjacent pixels. By locally spreading the matrix, the maximized Fisher ratio is given by(3)LlbT=δLlwT.The transformation matrix obtained by deformation calculation is(4)TLFDA=argmaxtrT∈Rd×rTLFDATLlwTLFDA−1TLFDATLlbTLFDA.LPPs reduce the adjacent data of the same kind via the transformation matrix,T. They effectively separate the adjacent data of different types while retaining the data’s local characteristics. Using the local retention and dimensionality reduction to preprocess the original HSI, the redundant information is eliminated. The distribution of similar classes is made more compact, the noise is reduced, and the classification accuracy is improved.


## 3.2. Multiscale Convolution

In this work, the use of a multiscale convolution kernel has two major advantages:The biggest advantage of multiscale convolution kernels is that different sizes of the convolution kernels can extract features of different scales from the HSIs. Thus, the filter can extract and learn richer information from them.When the CNN training model is used, training is achieved by learning the parameters of the filter (weights and offsets), i.e., by continuously learning the parameters of the filter to achieve an optimal value that is closest to the label. In this study, a multiscale convolution kernel has been used to enable every convolutional layer to have a variety of filters, thus achieving diversification of the weight and bias learning and a complete and effective extraction and learning of the useful information from the HSI.In computer vision models, multiscale inference methods are usually used for obtaining the best results. Generally, the finer details are efficiently predicted at larger sizes, the larger objects are efficiently predicted at smaller sizes, and the receiving field of the network can accurately understand the scene at smaller sizes. In contrast to the traditional multiscale structure, a novel multiscale dense convolution model (as shown in Figure2) has been proposed in this study. In particular, we apply the convolution process to extract the features in four sizes: 11 × 11, 7 × 7, 5 × 5, and 3 × 3. The corresponding calculation formulae are as follows:(5)Y1=ϕ∑i=0n1wij1∗xi1+bj1,Y2=addϕ∑i=0n1wij1∗xi1+bj1,ϕ∑i=0n2wij2∗xi2+bj2,Y3=ϕ∑i=0n3wij3∗xi3+bj3,Y4=addϕ∑i=0n1wij1∗xi1+bj1,ϕ∑i=0n2wij2∗xi2+bj2,ϕ∑i=0n3wij3∗xi3+bj3,Y5=addϕ∑i=0n1wij1∗xi1+bj1,ϕ∑i=0n2wij2∗xi2+bj2,ϕ∑i=0n3wij3∗xi3+bj3,ϕ∑i=0n4wij4∗xi4+bj4,where xi1, xi2, xi3, xi4, wij1, wij2, wij3, and wij4 represent the neurons and weights from the kernel of the multiscale convolutional layer and n represents the number of filters in CONV_i. As the output of the convolutional layer, Y is fed to the cross-attention-guided CP module.Figure 2
Schematic showing the multiscale layer that includes four different scale convolutions.Y1, Y2, Y3, Y4, and Y5 are the feature maps from the different scale convolutions.


## 3.3. Cross-Attention Mechanism

### 3.3.1. Attention Mechanism

In essence, deep learning’s attention mechanism is similar to the visual attention mechanism humans use when making choices. Both aim to filter out the key information, which is more helpful to the task they are dealing with, from a large amount of information, and suppress the useless information that is not relevant to the task at hand. The attention mechanism is prominent in two aspects: on the one hand, it can select the locally important information that is required to be focused on the overall input by itself; on the other hand, it can reasonably allocate a small amount of important information to the critical task objectives via computing resources. Because these two aspects, which enable the current attention mechanism network to be widely applied in image recognition, stand out, the attention mechanism can strengthen the characteristics of the local information. Further, the attention area position will change depending on the mission objectives; by understanding the target’s local information, the most useful information corresponding to the target can be put to good use. Figure3 shows the framework of the attention mechanism network. The output of the decoder part can be expressed as follows:(6)St=fst−1,yt−1,Ct,where St is the state output of the decoder at a time t, St−1 is the state output of the decoder at a time t−1, yt−1 is the label at a time t−1, and f is the dense layer.(7)Ct=∑j=1Txatjhj,where Ct is the output of the next state, hj is the output of the jth input in the decoder, and atj is the attention weight.(8)atj=expetj∑k=1Txexpetk,etj=gst−1,hj=v×tanhW×hj+u×St−1+b,where atj represents the degree of alignment between the current decoder and the jth input, which is the attention weight. g is used for calculating the relationship score between St−1 and hj. It can be seen that if the score is higher, the attention distribution is focused on the input.Figure 3
Schematic of the attention mechanism model.


### 3.3.2. Cross-Attention

Considering that the traditional attention mechanism assigns attention weight from only a single direction, it often leads to the loss of feature information. This study proposes a cross-attention mechanism, which constructs two novel weight distribution strategies: weight addition and maximization strategies. First, the cross-attention mechanism calculates the feature weight coefficients along with the horizontal and vertical directions. These two weight coefficients are added in order to enhance the feature, and the largest weight coefficient is obtained by applying the maximization strategy. Finally, the output of the two strategies is obtained via fusion. Figure4 shows a schematic diagram of the cross-attention mechanism.Figure 4
Schematic of the cross-attention mechanism.The equations for calculating the addition and maximization of weights are as follows:(9)Stj=∑j=1Txexpetj∑k=1Txexpetkhj,add=S1+S2,max=S1,S2,Cross_att=concatenateS1,S2,add,max,where S1 and S2 represent the weight coefficients of the horizontal and vertical attention mechanisms, respectively, add represents the addition of the weight coefficients, and max represents the maximization operation of the weight coefficients.


## 3.3.1. Attention Mechanism

In essence, deep learning’s attention mechanism is similar to the visual attention mechanism humans use when making choices. Both aim to filter out the key information, which is more helpful to the task they are dealing with, from a large amount of information, and suppress the useless information that is not relevant to the task at hand. The attention mechanism is prominent in two aspects: on the one hand, it can select the locally important information that is required to be focused on the overall input by itself; on the other hand, it can reasonably allocate a small amount of important information to the critical task objectives via computing resources. Because these two aspects, which enable the current attention mechanism network to be widely applied in image recognition, stand out, the attention mechanism can strengthen the characteristics of the local information. Further, the attention area position will change depending on the mission objectives; by understanding the target’s local information, the most useful information corresponding to the target can be put to good use. Figure3 shows the framework of the attention mechanism network. The output of the decoder part can be expressed as follows:(6)St=fst−1,yt−1,Ct,where St is the state output of the decoder at a time t, St−1 is the state output of the decoder at a time t−1, yt−1 is the label at a time t−1, and f is the dense layer.(7)Ct=∑j=1Txatjhj,where Ct is the output of the next state, hj is the output of the jth input in the decoder, and atj is the attention weight.(8)atj=expetj∑k=1Txexpetk,etj=gst−1,hj=v×tanhW×hj+u×St−1+b,where atj represents the degree of alignment between the current decoder and the jth input, which is the attention weight. g is used for calculating the relationship score between St−1 and hj. It can be seen that if the score is higher, the attention distribution is focused on the input.Figure 3
Schematic of the attention mechanism model.


## 3.3.2. Cross-Attention

Considering that the traditional attention mechanism assigns attention weight from only a single direction, it often leads to the loss of feature information. This study proposes a cross-attention mechanism, which constructs two novel weight distribution strategies: weight addition and maximization strategies. First, the cross-attention mechanism calculates the feature weight coefficients along with the horizontal and vertical directions. These two weight coefficients are added in order to enhance the feature, and the largest weight coefficient is obtained by applying the maximization strategy. Finally, the output of the two strategies is obtained via fusion. Figure4 shows a schematic diagram of the cross-attention mechanism.Figure 4
Schematic of the cross-attention mechanism.The equations for calculating the addition and maximization of weights are as follows:(9)Stj=∑j=1Txexpetj∑k=1Txexpetkhj,add=S1+S2,max=S1,S2,Cross_att=concatenateS1,S2,add,max,where S1 and S2 represent the weight coefficients of the horizontal and vertical attention mechanisms, respectively, add represents the addition of the weight coefficients, and max represents the maximization operation of the weight coefficients.


## 3.4. Covariance Pooling

The convolutional layer of a traditional CNN uses max/average pooling, and the fully connected layer can only capture the first-order information. Although the rectified linear unit (ReLU) introduces nonlinearity, it is only limited to the single-pixel level. We believe that the covariance matrix pooling (as shown in Figure5) can extract the features from HSIs with higher efficiency than the first-order statistics. If it is a set of features, its covariance matrix is(10)CP=1n−1∑i=1nfi−f¯fi−f¯T,where f¯=1/n∑i=1nfi. Only when the number of linear independent components in f1,f2,…,fn is greater than d, the resulting matrix is a symmetric positive definite (SPD) matrix. The covariance matrix must be SPD to use the SPD manifold network’s geometric structure to maintain the layer. However, even if the matrix is only positive semidefinite, it can be normalized by adding a multiple of the trace to the diagonal term of the covariance matrix as follows:(11)CP+=CP+λtraceCPI,where λ is the regular parameter and I is the identity matrix.Figure 5
Illustration of the concept of covariance pooling.


## 3.5. Feature Fusion

The multiscale convolution’s output features were spliced, and the CP that is guided by the cross-attention mechanism was used for obtaining the final deep features. The calculation equation for the same is written as follows:(12)O=addCP+Y1,CP+Cross_attY2,CP+Y3,CP+Cross_attY4,CP+Cross_attY5,where O represents the output of the CP module guided by the multiscale cross-attention mechanism.


## 4. Experiments

### 4.1. Evaluation Index

The results of the classification of HSIs require an evaluation of the classification ability of the MDCA-CP algorithm. For the same, evaluation indicators are required as a measurement standard. In this work, three evaluation indicators have been used for HSI classification: the kappa coefficient, the average accuracy (AA), and the overall accuracy (OA).


#### 4.1.1. OA

This evaluation index is expressed as the ratio of the number of correctly classified pixels to the total number of marked pixels. Ifn represents the number of categories of the image feature objects, Ni represents the number of pixels in the ith category, and hii represents the number of pixels that are correctly classified in the ith category, then the OA is expressed as(13)OA=∑i=1nhii∑i=1nNi.


#### 4.1.2. AA

This evaluation index is defined as follows. The ratio of the number of correctly classified pixels of each category to the category’s total number of pixels is calculated. The ratios of the overall categories are summed. The obtained sum is further divided by the overall categories, thus giving a number, which is the average classification accuracy. IfN represents the number of sample pixels that are to be tested in the overall training sample, n represents the number of categories, and hii represents the number of correctly classified pixels of the ith category, then AA is expressed as(14)AA=1n∑i=1nhiiNi.


#### 4.1.3. Kappa

This coefficient is a performance index used for measuring the accuracy of classification via the classification confusion matrix and is used for consistency testing. It can be expressed as follows:(15)kappa=Po−Pe1−Pe.


### 4.2. Datasets

This paper will test the proposed MDCA-CP on three benchmark datasets: IP (Indian Pines), PU (University of Pavia), and SA (Salina) to verify its effectiveness. An AVIRIS sensor over northwestern Indiana took the IP dataset. It comprises 145 pixels × 145 pixels in the spatial domain. The spectral domain is composed of 224 spectral reflectance bands, and the wavelength range is 0.4–2.45μm. Among them, the available ground truth is classified into 16 categories. In this paper’s experiment, 20 water absorption bands are removed, and the final image size is 145 pixels × 145 pixels × 200 pixels. A ROSIS sensor took the PU dataset over northern Italy. It consists of 10 pixels × 340 pixels in the spatial domain; after removing the water absorption band, the spectral domain consists of 103 spectral bands, and the spectral coverage ranges from 430 to 860 nm. Among them, the ground truth is classified into 9 categories, and the size of the image used in the experiment of this article is 610 × 340 × 103.The SA dataset was taken over California by the AVIRIS sensor. It has 224 bands in the spectral domain and 512 pixels × 217 pixels in the spatial domain. It also has a high spatial resolution (3.7 m pixels).Features. After removing the 20 absorbent bands, the experimental image’s size is 512 pixels × 217 pixels × 204 pixels, and the available ground truth is classified into 16 categories. Tables 1–3 describe the sample numbers of these three datasets in detail.Table 1
Subject category distribution of the IP dataset.
TypeSpeciesSamplesC1Alfalfa46C2Corn-notill1428C3Corn-mintill830C4Corn237C5Grass-pasture483C6Grass-trees730C7Grass-pasture-mowed28C8Hay-windrowed478C9Oats20C10Soybean-notill972C11Soybean-mintill2455C12Soybean-clean593C13Wheat205C14Woods1265C15Buildings-grass-trees-drives386C16Stone-steel-towers93Table 2
Subject category distribution of the PU dataset.
TypeClassSamplesC1Asphalt6631C2Meadows18649C3Gravel2099C4Trees3064C5Painted metal sheets1345C6Bare soil5029C7Bitumen1330C8Self-blocking bricks3682C9Shadows947Table 3
Subject category distribution of the SV dataset.
TypeClassSamplesC1Brocoli_green_weeds_12009C2Brocoli_green_weeds_23726C3Fallow1976C4Fallow_rough_plow1394C5Fallow_smooth2678C6Stubble3959C7Celery3579C8Grapes_untrained11271C9Soil_vinyard_develop6203C10Corn_senesced_green_weeds3278C11Lettuce_romaine_4wk1068C12Lettuce_romaine_5wk1927C13Lettuce_romaine_6wk916C14Lettuce_romaine_7wk1070C15Vinyard_untrained7268C16Vinyard_vertical_trellis1807


### 4.3. Hyperparameter Setting

The MDCA-CP proposed in this paper is based on Python language and Keras deep learning framework. The experimental environment was the Windows 10 operating system, 16 GB RAM, and NVIDIA GeForce GTX 1080 8 GB GPU. In order to prevent the deviation caused by different training samples, the average value of more than 20 experimental results under the same conditions was analyzed in this paper. In this model, the random gradient descent method is adopted to update the weight. The learning rate is 0.01, the proportion of disconnected neurons in the dropout layer of the whole connection is set as 0.5, and the activation function is ReLU. In this paper, the bilateral fusion block network was trained for small-batch gradient descent. The number of training samples was set as 1000, and the EPOCH was set as 600, and Table4 shows the detailed hyperparameter settings.Table 4
Experimental parameter settings.
TypeHyperparameterAdamLearning rate = 0.01β1 = 0.9β2 = 0.999Epsilon = 1e-08Decay = 3e-8


### 4.4. Experimental Results Obtained from Different Methods

To verify the validity and correctness of our proposed method, a comparison of the results obtained using MDCA-CP was done with those obtained using AlexNet [23], ResNet [33], DenseNet [34], PRAN [35], FSSFNet [36], and SAGP [37]. In order to ensure fairness in the experiment, all hyperparameters in the comparison network were set to the same value. The results obtained by applying the different models on the three datasets are given in Tables 5–7.Table 5
Classification results of different methods on the IP data (%).
Methods5%10%15%OAAAKappaOAAAKappaOAAAKappaAlexNet [23]68.6956.2964.0574.3265.1770.9481.7679.4479.70ResNet [33]70.6769.6966.0478.4079.2275.1083.3380.7680.95DenseNet [34]71.2367.5566.6678.3775.2975.8784.6681.3782.58PRAN [35]72.4573.3769.4877.6673.2874.4782.7276.5180.28FSSFNet [36]73.7567.9569.9878.6671.2675.4782.6174.4880.05SAGP [37]73.4976.5873.6178.3680.8976.7281.5986.5082.89MDCA-CP76.8579.1173.7282.5681.8780.0387.3287.6585.55Table 6
Classification results of different methods on the PU data (%).
Methods1%5%10%OAAAKappaOAAAKappaOAAAKappaAlexNet [23]87.1584.2482.6692.8191.7790.4392.8593.4691.32ResNet [33]84.0682.3877.5990.8191.3287.7694.1594.7893.19DenseNet [34]82.3280.6175.6990.2589.4987.0892.2793.4691.21PRAN [35]89.2788.6185.6393.4592.3291.4894.7193.5593.02FSSFNet [36]86.6384.2382.1193.7292.3591.6594.3792.6592.52SAGP [37]84.3283.8478.8391.0989.7588.1493.7392.9791.65MDCA-CP89.4087.9786.8193.5591.9192.2194.9993.0293.11Table 7
Classification results of different methods on the SV data (%).
Methods1%5%10%OAAAKappaOAAAKappaOAAAKappaAlexNet [23]90.8893.7289.7794.0596.6493.4294.2695.6795.29ResNet [33]87.2192.0785.8891.7095.1590.7493.5296.7393.11DenseNet [34]85.5290.8184.1291.0394.3690.0593.1496.1292.17PRAN [35]78.8676.1776.4990.2888.1789.1691.8189.3291.17FSSFNet [36]91.0494.7490.0293.5696.6992.8295.8598.0395.37SAGP [37]90.8794.6889.8292.6495.6991.8194.7697.4594.16MDCA-CP91.4594.8489.9894.5197.1193.2496.1598.1196.21


#### 4.4.1. Experimental Results Obtained Using the IP Dataset

We randomly selected 5% of the training samples from the IP dataset for training and the remaining 95% samples for testing. It can be seen from Table5 and Figure 6 that the AlexNet model has the worst classification performance, with considerable noise in its classification graph. This is because its model is not deep enough to extract representative discriminant features. It fails to make corresponding optimization adjustments for overfitting and resolution decline in the training process. In the other classification models, the model’s classification results with the attention mechanism algorithm are better than those obtained by other methods. In addition, compared with PRAN, SAGP has a better classification performance, which is due to the extreme imbalance of category samples in the IP dataset. The attention mechanism overcomes this disadvantage, so the MDCA-CP proposed in this paper achieves the optimal results.Figure 6
Classification results on the IP dataset using 5% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


#### 4.4.2. Experimental Results Obtained Using the PU Dataset

From Table6 and Figure 7, it can be seen that, in the Pavia University dataset, among the accuracy rates obtained by various methods, the MDCA-CP method proposed in this work exhibits the highest classification accuracy. As observed in Figure 7, the method proposed in this work has a larger number of correctly classified pixels in the classification map and has less noise than the other methods in some areas. This is because SAGP contains an attention mechanism network that focuses on the features that play an important role in the current task goal by strengthening the local features and selects the current state from many features. The network model can use the analysis of the key features to replace the overall image features analysis and enhance the weight of the features related to the current task goal. The targeted features will not help the classification as they are weakened. When performing the classification process, attention can be focused on finding the useful features associated with the current output to obtain better feature representations for different categories, which helps the final classification performance. From the above description, we can see that the method proposed in this study is better than the other methods.Figure 7
Classification results on the PU dataset using 1% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


#### 4.4.3. Experimental Results Obtained Using the SV Dataset

For the SV dataset, this paper randomly selects 1%, 5%, and 10% of training samples, and the remaining samples are used as test samples. Table7 and Figure 8 show the comparative experimental results. Similarly, in terms of visual effects, the ground object classification map displayed on the SV dataset by the method proposed in this paper has the slightest noise. In quantitative analysis, the OA on the SA data set of the MDCA-CP model reaches 96.15%, which is higher than other comparison methods on OA, KA, and AA. The proposed MDCA-CP model can effectively identify most of the ground features and perform better in the SV dataset.Figure 8
Classification results on the SV dataset using 1% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


### 4.5. Results of Ablation Experiments on Different Weighting Strategies

Table8 shows the experimental results obtained by using different weight allocation strategies. From the table, it can be seen that the combined addition and maximization strategy is significantly better than the individual strategies of addition only or maximization only. The combined strategy enhances the model classification ability, proving that the two strategies of MDCA-CP are effective. Further, the weight addition strategy is superior to the weight maximization strategy, thus proving that only using the maximization strategy will lead to a loss of some features. Figures 9–11 also show the limit pie chart of each indicator on the three datasets with different weighting strategies.Table 8
Results of ablation experiments on different weighting strategies (%).
ModulesIPPUSVOAAAKappaOAAAKappaOAAAKappaAdd74.7775.4871.0192.0290.5889.2092.4894.8592.99Max74.1274.3670.2591.5889.7489.0291.5294.4491.85Add + Max (MDCA-CP)76.8579.1173.7293.5591.9192.2194.5197.1193.24Figure 9
Classification results on the IP dataset using different weighting strategies. (a) OA. (b) AA. (c) Kappa.
(a)(b)(c)Figure 10
Classification results on the PU dataset using different weighting strategies. (a) OA. (b) AA. (c) Kappa.
(a)(b)(c)Figure 11
Classification results on the SV dataset using different weighting strategies. (a) OA. (b) AA. (c) Kappa.
(a)(b)(c)


### 4.6. Results of Ablation Experiments on Different Submodules

This section presents the results of the ablation experiments on the cross-attention module and the CP module. Table9 and Figure 12 show the corresponding experimental results from which it is observed that a combination of the two models (MDCA-CP) provides superior results. Similarly, we found that the cross-attention module has better performance than the covariance pooling module, which further shows the importance of feature mining in hyperspectral image classification. The MDCA module can effectively select the most representative features. In contrast, the covariance module is better than the baseline model, proving that data mining from HSIs and extracting second-order features are also effective. This further proves the superiority of our MDCA-CP model. Figures 13–15 also show the histograms of the indicators of the different submodules on the three datasets.Table 9
Results of ablation experiments on different submodules (%).
ModulesIPPUSVOAAAKappaOAAAKappaOAAAKappaNo-CrossAttention74.7876.5472.2490.8189.1790.1091.7894.2591.36No-CovariancePooling76.1371.1371.7793.2590.6591.1193.8995.2592.15MDCA-CP76.8579.1173.7293.5591.9192.2194.5197.1193.24Figure 12
Classification results on the IP dataset using different submodules. (a) Ground truth. (b) No MDCA. (c) No CP. (d) MDCA-CP.
(a)(b)(c)(d)Figure 13
Classification results on the IP dataset using different submodules.Figure 14
Classification results on the PU dataset using different submodules.Figure 15
Classification results on the SV dataset using different submodules.


## 4.1. Evaluation Index

The results of the classification of HSIs require an evaluation of the classification ability of the MDCA-CP algorithm. For the same, evaluation indicators are required as a measurement standard. In this work, three evaluation indicators have been used for HSI classification: the kappa coefficient, the average accuracy (AA), and the overall accuracy (OA).


### 4.1.1. OA

This evaluation index is expressed as the ratio of the number of correctly classified pixels to the total number of marked pixels. Ifn represents the number of categories of the image feature objects, Ni represents the number of pixels in the ith category, and hii represents the number of pixels that are correctly classified in the ith category, then the OA is expressed as(13)OA=∑i=1nhii∑i=1nNi.


### 4.1.2. AA

This evaluation index is defined as follows. The ratio of the number of correctly classified pixels of each category to the category’s total number of pixels is calculated. The ratios of the overall categories are summed. The obtained sum is further divided by the overall categories, thus giving a number, which is the average classification accuracy. IfN represents the number of sample pixels that are to be tested in the overall training sample, n represents the number of categories, and hii represents the number of correctly classified pixels of the ith category, then AA is expressed as(14)AA=1n∑i=1nhiiNi.


### 4.1.3. Kappa

This coefficient is a performance index used for measuring the accuracy of classification via the classification confusion matrix and is used for consistency testing. It can be expressed as follows:(15)kappa=Po−Pe1−Pe.


## 4.1.1. OA

This evaluation index is expressed as the ratio of the number of correctly classified pixels to the total number of marked pixels. Ifn represents the number of categories of the image feature objects, Ni represents the number of pixels in the ith category, and hii represents the number of pixels that are correctly classified in the ith category, then the OA is expressed as(13)OA=∑i=1nhii∑i=1nNi.


## 4.1.2. AA

This evaluation index is defined as follows. The ratio of the number of correctly classified pixels of each category to the category’s total number of pixels is calculated. The ratios of the overall categories are summed. The obtained sum is further divided by the overall categories, thus giving a number, which is the average classification accuracy. IfN represents the number of sample pixels that are to be tested in the overall training sample, n represents the number of categories, and hii represents the number of correctly classified pixels of the ith category, then AA is expressed as(14)AA=1n∑i=1nhiiNi.


## 4.1.3. Kappa

This coefficient is a performance index used for measuring the accuracy of classification via the classification confusion matrix and is used for consistency testing. It can be expressed as follows:(15)kappa=Po−Pe1−Pe.


## 4.2. Datasets

This paper will test the proposed MDCA-CP on three benchmark datasets: IP (Indian Pines), PU (University of Pavia), and SA (Salina) to verify its effectiveness. An AVIRIS sensor over northwestern Indiana took the IP dataset. It comprises 145 pixels × 145 pixels in the spatial domain. The spectral domain is composed of 224 spectral reflectance bands, and the wavelength range is 0.4–2.45μm. Among them, the available ground truth is classified into 16 categories. In this paper’s experiment, 20 water absorption bands are removed, and the final image size is 145 pixels × 145 pixels × 200 pixels. A ROSIS sensor took the PU dataset over northern Italy. It consists of 10 pixels × 340 pixels in the spatial domain; after removing the water absorption band, the spectral domain consists of 103 spectral bands, and the spectral coverage ranges from 430 to 860 nm. Among them, the ground truth is classified into 9 categories, and the size of the image used in the experiment of this article is 610 × 340 × 103.The SA dataset was taken over California by the AVIRIS sensor. It has 224 bands in the spectral domain and 512 pixels × 217 pixels in the spatial domain. It also has a high spatial resolution (3.7 m pixels).Features. After removing the 20 absorbent bands, the experimental image’s size is 512 pixels × 217 pixels × 204 pixels, and the available ground truth is classified into 16 categories. Tables 1–3 describe the sample numbers of these three datasets in detail.Table 1
Subject category distribution of the IP dataset.
TypeSpeciesSamplesC1Alfalfa46C2Corn-notill1428C3Corn-mintill830C4Corn237C5Grass-pasture483C6Grass-trees730C7Grass-pasture-mowed28C8Hay-windrowed478C9Oats20C10Soybean-notill972C11Soybean-mintill2455C12Soybean-clean593C13Wheat205C14Woods1265C15Buildings-grass-trees-drives386C16Stone-steel-towers93Table 2
Subject category distribution of the PU dataset.
TypeClassSamplesC1Asphalt6631C2Meadows18649C3Gravel2099C4Trees3064C5Painted metal sheets1345C6Bare soil5029C7Bitumen1330C8Self-blocking bricks3682C9Shadows947Table 3
Subject category distribution of the SV dataset.
TypeClassSamplesC1Brocoli_green_weeds_12009C2Brocoli_green_weeds_23726C3Fallow1976C4Fallow_rough_plow1394C5Fallow_smooth2678C6Stubble3959C7Celery3579C8Grapes_untrained11271C9Soil_vinyard_develop6203C10Corn_senesced_green_weeds3278C11Lettuce_romaine_4wk1068C12Lettuce_romaine_5wk1927C13Lettuce_romaine_6wk916C14Lettuce_romaine_7wk1070C15Vinyard_untrained7268C16Vinyard_vertical_trellis1807


## 4.3. Hyperparameter Setting

The MDCA-CP proposed in this paper is based on Python language and Keras deep learning framework. The experimental environment was the Windows 10 operating system, 16 GB RAM, and NVIDIA GeForce GTX 1080 8 GB GPU. In order to prevent the deviation caused by different training samples, the average value of more than 20 experimental results under the same conditions was analyzed in this paper. In this model, the random gradient descent method is adopted to update the weight. The learning rate is 0.01, the proportion of disconnected neurons in the dropout layer of the whole connection is set as 0.5, and the activation function is ReLU. In this paper, the bilateral fusion block network was trained for small-batch gradient descent. The number of training samples was set as 1000, and the EPOCH was set as 600, and Table4 shows the detailed hyperparameter settings.Table 4
Experimental parameter settings.
TypeHyperparameterAdamLearning rate = 0.01β1 = 0.9β2 = 0.999Epsilon = 1e-08Decay = 3e-8


## 4.4. Experimental Results Obtained from Different Methods

To verify the validity and correctness of our proposed method, a comparison of the results obtained using MDCA-CP was done with those obtained using AlexNet [23], ResNet [33], DenseNet [34], PRAN [35], FSSFNet [36], and SAGP [37]. In order to ensure fairness in the experiment, all hyperparameters in the comparison network were set to the same value. The results obtained by applying the different models on the three datasets are given in Tables 5–7.Table 5
Classification results of different methods on the IP data (%).
Methods5%10%15%OAAAKappaOAAAKappaOAAAKappaAlexNet [23]68.6956.2964.0574.3265.1770.9481.7679.4479.70ResNet [33]70.6769.6966.0478.4079.2275.1083.3380.7680.95DenseNet [34]71.2367.5566.6678.3775.2975.8784.6681.3782.58PRAN [35]72.4573.3769.4877.6673.2874.4782.7276.5180.28FSSFNet [36]73.7567.9569.9878.6671.2675.4782.6174.4880.05SAGP [37]73.4976.5873.6178.3680.8976.7281.5986.5082.89MDCA-CP76.8579.1173.7282.5681.8780.0387.3287.6585.55Table 6
Classification results of different methods on the PU data (%).
Methods1%5%10%OAAAKappaOAAAKappaOAAAKappaAlexNet [23]87.1584.2482.6692.8191.7790.4392.8593.4691.32ResNet [33]84.0682.3877.5990.8191.3287.7694.1594.7893.19DenseNet [34]82.3280.6175.6990.2589.4987.0892.2793.4691.21PRAN [35]89.2788.6185.6393.4592.3291.4894.7193.5593.02FSSFNet [36]86.6384.2382.1193.7292.3591.6594.3792.6592.52SAGP [37]84.3283.8478.8391.0989.7588.1493.7392.9791.65MDCA-CP89.4087.9786.8193.5591.9192.2194.9993.0293.11Table 7
Classification results of different methods on the SV data (%).
Methods1%5%10%OAAAKappaOAAAKappaOAAAKappaAlexNet [23]90.8893.7289.7794.0596.6493.4294.2695.6795.29ResNet [33]87.2192.0785.8891.7095.1590.7493.5296.7393.11DenseNet [34]85.5290.8184.1291.0394.3690.0593.1496.1292.17PRAN [35]78.8676.1776.4990.2888.1789.1691.8189.3291.17FSSFNet [36]91.0494.7490.0293.5696.6992.8295.8598.0395.37SAGP [37]90.8794.6889.8292.6495.6991.8194.7697.4594.16MDCA-CP91.4594.8489.9894.5197.1193.2496.1598.1196.21


### 4.4.1. Experimental Results Obtained Using the IP Dataset

We randomly selected 5% of the training samples from the IP dataset for training and the remaining 95% samples for testing. It can be seen from Table5 and Figure 6 that the AlexNet model has the worst classification performance, with considerable noise in its classification graph. This is because its model is not deep enough to extract representative discriminant features. It fails to make corresponding optimization adjustments for overfitting and resolution decline in the training process. In the other classification models, the model’s classification results with the attention mechanism algorithm are better than those obtained by other methods. In addition, compared with PRAN, SAGP has a better classification performance, which is due to the extreme imbalance of category samples in the IP dataset. The attention mechanism overcomes this disadvantage, so the MDCA-CP proposed in this paper achieves the optimal results.Figure 6
Classification results on the IP dataset using 5% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


### 4.4.2. Experimental Results Obtained Using the PU Dataset

From Table6 and Figure 7, it can be seen that, in the Pavia University dataset, among the accuracy rates obtained by various methods, the MDCA-CP method proposed in this work exhibits the highest classification accuracy. As observed in Figure 7, the method proposed in this work has a larger number of correctly classified pixels in the classification map and has less noise than the other methods in some areas. This is because SAGP contains an attention mechanism network that focuses on the features that play an important role in the current task goal by strengthening the local features and selects the current state from many features. The network model can use the analysis of the key features to replace the overall image features analysis and enhance the weight of the features related to the current task goal. The targeted features will not help the classification as they are weakened. When performing the classification process, attention can be focused on finding the useful features associated with the current output to obtain better feature representations for different categories, which helps the final classification performance. From the above description, we can see that the method proposed in this study is better than the other methods.Figure 7
Classification results on the PU dataset using 1% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


### 4.4.3. Experimental Results Obtained Using the SV Dataset

For the SV dataset, this paper randomly selects 1%, 5%, and 10% of training samples, and the remaining samples are used as test samples. Table7 and Figure 8 show the comparative experimental results. Similarly, in terms of visual effects, the ground object classification map displayed on the SV dataset by the method proposed in this paper has the slightest noise. In quantitative analysis, the OA on the SA data set of the MDCA-CP model reaches 96.15%, which is higher than other comparison methods on OA, KA, and AA. The proposed MDCA-CP model can effectively identify most of the ground features and perform better in the SV dataset.Figure 8
Classification results on the SV dataset using 1% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


## 4.4.1. Experimental Results Obtained Using the IP Dataset

We randomly selected 5% of the training samples from the IP dataset for training and the remaining 95% samples for testing. It can be seen from Table5 and Figure 6 that the AlexNet model has the worst classification performance, with considerable noise in its classification graph. This is because its model is not deep enough to extract representative discriminant features. It fails to make corresponding optimization adjustments for overfitting and resolution decline in the training process. In the other classification models, the model’s classification results with the attention mechanism algorithm are better than those obtained by other methods. In addition, compared with PRAN, SAGP has a better classification performance, which is due to the extreme imbalance of category samples in the IP dataset. The attention mechanism overcomes this disadvantage, so the MDCA-CP proposed in this paper achieves the optimal results.Figure 6
Classification results on the IP dataset using 5% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


## 4.4.2. Experimental Results Obtained Using the PU Dataset

From Table6 and Figure 7, it can be seen that, in the Pavia University dataset, among the accuracy rates obtained by various methods, the MDCA-CP method proposed in this work exhibits the highest classification accuracy. As observed in Figure 7, the method proposed in this work has a larger number of correctly classified pixels in the classification map and has less noise than the other methods in some areas. This is because SAGP contains an attention mechanism network that focuses on the features that play an important role in the current task goal by strengthening the local features and selects the current state from many features. The network model can use the analysis of the key features to replace the overall image features analysis and enhance the weight of the features related to the current task goal. The targeted features will not help the classification as they are weakened. When performing the classification process, attention can be focused on finding the useful features associated with the current output to obtain better feature representations for different categories, which helps the final classification performance. From the above description, we can see that the method proposed in this study is better than the other methods.Figure 7
Classification results on the PU dataset using 1% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


## 4.4.3. Experimental Results Obtained Using the SV Dataset

For the SV dataset, this paper randomly selects 1%, 5%, and 10% of training samples, and the remaining samples are used as test samples. Table7 and Figure 8 show the comparative experimental results. Similarly, in terms of visual effects, the ground object classification map displayed on the SV dataset by the method proposed in this paper has the slightest noise. In quantitative analysis, the OA on the SA data set of the MDCA-CP model reaches 96.15%, which is higher than other comparison methods on OA, KA, and AA. The proposed MDCA-CP model can effectively identify most of the ground features and perform better in the SV dataset.Figure 8
Classification results on the SV dataset using 1% training samples. (a) Ground truth. (b) AlexNet. (c) ResNet. (d) DenseNet. (e) PRAN. (f) FSSFNet. (g) SAGP. (h) MDCA-CP.


## 4.5. Results of Ablation Experiments on Different Weighting Strategies

Table8 shows the experimental results obtained by using different weight allocation strategies. From the table, it can be seen that the combined addition and maximization strategy is significantly better than the individual strategies of addition only or maximization only. The combined strategy enhances the model classification ability, proving that the two strategies of MDCA-CP are effective. Further, the weight addition strategy is superior to the weight maximization strategy, thus proving that only using the maximization strategy will lead to a loss of some features. Figures 9–11 also show the limit pie chart of each indicator on the three datasets with different weighting strategies.Table 8
Results of ablation experiments on different weighting strategies (%).
ModulesIPPUSVOAAAKappaOAAAKappaOAAAKappaAdd74.7775.4871.0192.0290.5889.2092.4894.8592.99Max74.1274.3670.2591.5889.7489.0291.5294.4491.85Add + Max (MDCA-CP)76.8579.1173.7293.5591.9192.2194.5197.1193.24Figure 9
Classification results on the IP dataset using different weighting strategies. (a) OA. (b) AA. (c) Kappa.
(a)(b)(c)Figure 10
Classification results on the PU dataset using different weighting strategies. (a) OA. (b) AA. (c) Kappa.
(a)(b)(c)Figure 11
Classification results on the SV dataset using different weighting strategies. (a) OA. (b) AA. (c) Kappa.
(a)(b)(c)


## 4.6. Results of Ablation Experiments on Different Submodules

This section presents the results of the ablation experiments on the cross-attention module and the CP module. Table9 and Figure 12 show the corresponding experimental results from which it is observed that a combination of the two models (MDCA-CP) provides superior results. Similarly, we found that the cross-attention module has better performance than the covariance pooling module, which further shows the importance of feature mining in hyperspectral image classification. The MDCA module can effectively select the most representative features. In contrast, the covariance module is better than the baseline model, proving that data mining from HSIs and extracting second-order features are also effective. This further proves the superiority of our MDCA-CP model. Figures 13–15 also show the histograms of the indicators of the different submodules on the three datasets.Table 9
Results of ablation experiments on different submodules (%).
ModulesIPPUSVOAAAKappaOAAAKappaOAAAKappaNo-CrossAttention74.7876.5472.2490.8189.1790.1091.7894.2591.36No-CovariancePooling76.1371.1371.7793.2590.6591.1193.8995.2592.15MDCA-CP76.8579.1173.7293.5591.9192.2194.5197.1193.24Figure 12
Classification results on the IP dataset using different submodules. (a) Ground truth. (b) No MDCA. (c) No CP. (d) MDCA-CP.
(a)(b)(c)(d)Figure 13
Classification results on the IP dataset using different submodules.Figure 14
Classification results on the PU dataset using different submodules.Figure 15
Classification results on the SV dataset using different submodules.


## 5. Conclusion

This paper presents the data dimensionality reduction and feature extraction and presents the novel MDCA-CP model for HSI scene classification. Multiscale convolution has been proven to effectively detect the subtle changes between the pixels in the local area of spatial and spectral dimensions of an HSI, and it can be applied to the feature extraction of hyperspectral data of complex types and structures. Traditional algorithms assign the attention weights in a one-way manner, thus resulting in the loss of feature information. This work’s dense cross-attention mechanism can jointly distribute the attention weights horizontally and vertically. As a result, it can efficiently mine the most representative features of the hyperspectral data. A further enhancement in mining the features can be achieved by employing the weight addition and maximization strategies. This paper also proves the effectiveness and superiority of the MDCA-CP algorithm by comparing results obtained from different models and conducting ablation experiments.



---

*Source: 9962057-2021-05-04.xml*