# Convolution-Transformer Adaptive Fusion Network for Hyperspectral Image Classification
###### Abstract
Several studies have attempted to improve the CNN models by directly extending the receptive field of the convolution kernel, including the use of dilated convolutions [23] and the construction of multi-scale feature pyramids [24]. Recent studies also introduced the attentional mechanism to enhance the useful components in the features while suppressing the useless ones, such as the spectral attention [25], spatial attention [26], and spectral-spatial attention [27]. Nevertheless, the above-mentioned methods do not fully overcome the limitations of CNNs, as they depend strictly on convolution operations, which are incapable of modeling long-term dependencies [28]. In recent years, the emergence of the Transformer has provided a new means for HSI classification. Transformer is a new neural network architecture consisting of a multi-head self-attention (MSA) module and a feed-forward neural network [29]. By introducing the MSA module, Transformer can effectively capture long-term dependencies. [PERSON] et al. [30] proposed a self-attention Transformer network (SATNet) for HSI classification. SATNet employs Transformer encoders to extract image features and uses a multilevel residual structure to connect multiple encoder blocks to solve the vanishing gradient and over-fitting problems. [PERSON] et al. [31] proposed an encoder-decoder network that fuses local-global spatial attention and spectral attention (FSSANet) for HSI classification. FSSANet introduces spectral attention into the Swin Transformer encoder [32] to encode the rich spectral-spatial information of HSI and therefore improves the classification accuracy. Although the application of Transformer as a backbone network for feature extraction has a good performance in HSI classification, it still faces two problems. First, Transformer needs to convert images to low-dimensional patch embeddings, which destroys the internal structure of the images and increases the requirement of quantity of training data to learn the unique properties of the images [33]. Secondly, Transformer cannot learn the correlation between different pixels within a patch, and it is difficult to capture local high-frequency information [34]. After reviewing the CNN-based and Transformer-based models, it was found that they complement each other, and combining use of them offers opportunities to enhance the modeling of both local high-frequency information and global long-term dependencies [35]. [PERSON] et al. [36] proposed a spectral-spatial feature tokenization Transformer (SSFTT), which uses convolution layers to extract shallow spectral-spatial features and the Transformerencoder to capture deep semantic information. The extracted HSI spectral-spatial semantic features from shallow to deep were fused to improve classification accuracy. [PERSON] et al. [37] proposed a two branch HSI classification framework based on three-dimensional CNN and bottleneck spatial-spectral Transformer (B2 ST). In this framework, both branches use a combination of shallow CNN and deep Transformer. One branch is used to extract spatial local-global joint features, and the other is focused on extracting spectral local-global joint features. The fused spectral-spatial features can express the local global semantic information, thus achieving outstanding classification performance. Although the existing CNN-Transformer hybrid models have made progress in HSI classification, they have at least two disadvantages. First, existing methods integrate convolution and Transformer through artificially specified strategies, which are empirical and might lead to poor generalization ability. Second, most of these models adopt the image-wise classification network based on \"encoder-label\" structure, which make predictions through the features of the last stage and cannot make full use of the information obtained from the other stages. As a result, the \"encoder-label\" structure might yield incorrect results, e.g., omission of fine-grained objects and confusion between similar objects in HSI of complex scenes [21,38].