# Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification
###### Abstract  
Hyperspectral image (HSI) classification is an important but challenging topic in the field of remote sensing and earth observation. By coupling the advantages of convolutional neural network (CNN) and Transformer model, the CNN-Transformer hybrid model can extract local and global features simultaneously and has achieved outstanding performance in HSI classification. However, most of the existing CNN-Transformer hybrid models use artificially specified hybrid strategies, which have poor generalization ability and are difficult to meet the requirements of recognizing fine-grained objects in HSI of complex scenes. To overcome this problem, we proposed a convolution-Transformer adaptive fusion network (CTAFNet) for pixel-wise HSI classification. A local-global fusion feature extraction unit, called the convolution-Transformer adaptive fusion kernel, was designed and integrated into the CTAFNet. The kernel captures the local high-frequency features using a convolution module and extracts the global and sequential low-frequency information using a Transformer module. We developed an adaptive feature fusion strategy to fuse the local high-frequency and global low-frequency features to obtain a robust and discriminative representation of the HSI data. An encoder-decoder structure was adopted in the CTAFNet to improve the flow of fused local-global information between different stages, thus ensuring the generalization ability of the model. Experimental results conducted on three large-scale and challenging HSI datasets demonstrate that the proposed network is superior to nine state-of-the-art approaches. We highlighted the effectiveness of adaptive CNN-Transformer hybrid strategy in HSI classification. deep learning; hyperspectral image classification; convolutional neural networks; transformer; hybrid strategy; feature fusion +
Footnote †: journal:  
feature transform (SIFT) [11], local binary patterns (LBP) [12], and extended morphological profile (EMP) [13] were used to extract spatial features. On the basis of artificially designed features, traditional machine learning methods, such as support vector machine (SVM) [14], random forest (RF) [15], and extreme learning machine (ELM) [16] were used to classify HSI pixels into different types. The performance of these methods largely depends on the quality of the manually designed features. Generalization and robustness of these methods are generally poor because the features are designed for specific tasks, and the feature extraction requires complex processes of parameter tuning. Different from traditional methods that rely on artificially designed features, deep learning-based methods can automatically learn multi-level nonlinear features, which are conducive to analyzing the inherent characteristics of HSI [7]. In recent years, as a representative deep learning model, the convolutional neural network (CNN) has made milestone progress in HSI classification. Relevant scholars have proposed a variety of CNN-based classification models, which can be divided into three categories according to the types of extracted features: (1) spectral CNN, which uses 1D CNN [17] to extract the spectral features of HSI; (2) spatial CNN, which uses two-dimensional CNN [18] to extract the spatial features from HSI after dimension reduction; (3) spectral-spatial CNN, which uses three-dimensional convolution [19] or dual branch network [20] to simultaneously extract spectral-spatial joint features of HSI. Although these CNN models have yielded satisfactory results in specific applications, they generally face the following two challenges: (1) due to the limited receptive field, they can hardly capture low-frequency signals, which provide global information (e.g., global shapes and structures) [21]; (2) the quality of the extracted high-frequency signals (e.g., local edges and texture) needs to be improved [22].